Method and apparatus for suggesting/disambiguation query terms based upon usage patterns observed

ABSTRACT

The invention comprises a set of complementary techniques that dramatically improve enterprise search and navigation results. The core of the invention is an expertise or knowledge index, called UseRank that tracks the behavior of website visitors. The expertise-index is designed to focus on the four key discoveries of enterprise attributes: Subject Authority, Work Patterns, Content Freshness, and Group Know-how. The invention produces useful, timely, cross-application, expertise-based search and navigation results. In contrast, traditional Information Retrieval technologies such as inverted index, NLP, or taxonomy tackle the same problem with an opposite set of attributes than what the enterprise needs: Content Population, Word Patterns, Content Existence, and Statistical Trends. Overall, the invention emcompasses Baynote Search—a enhancement over existing IR searches, Baynote Guide —a set of community-driven navigations, and Baynote Insights—aggregated views of visitor interests and trends and content gaps.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a divisional which claims priority from U.S.application Ser. No. 11/319,928 filed Dec. 27, 2005, which claimspriority from U.S. provisional Application No. 60/640,872 filed Dec. 29,2004, which is incorporated by reference in its entirety, for allpurposes, herein.

BACKGROUND OF THE INVENTION

1. Technical Field

The invention relates to electronic access to information. Moreparticularly, the invention relates to a method and apparatus foridentifying, extracting, capturing, and leveraging expertise andknowledge.

2. Description of the Prior Art

For years enterprises have struggled with ineffective search techniques.Compared to what is available on the public Web via services such asGoogle and Yahoo, there remains a dearth of highly relevant searchsolutions for content within the enterprise. With dozens to hundreds ofindependent application and repository information silos in eachcompany, finding critical business information costs business hundredsof billions of dollars each year [source: A. T. Kearney]. Today, CIOsand business executives are revisiting enterprise search as one of thetop business/IT challenges for the next few years.

Enterprise search needs are poorly served. Various search technologieshave been developed to attack the challenge of searching the Web,searching individual user's computers (PC/desktop), and searching theinternal business documents (enterprise). Each of these approaches areunique, but none provide an adequate solution for the enterprise.

PC (Desktop) Search

PC or desktop search can be compared with finding stuff in your messygarage. You know you have it somewhere but just cannot find it. So tolocate is the only goal. And when you do find it, you are the sole judgeto decide if you have indeed found the right content or document becauseyou collected or wrote the content in the first place. You are the onlyexpert and authority that matters.

Traditional PC search from Microsoft is based on parsing a file at thetime of search. It is slow and can only find things in isolated places,such as file folders or email directories. The latest PC searchintroduces inverted index technology from Google, soon to be availablealso from Yahoo, Ask Jeeves, and Microsoft. They start to solve thespeed and silo problems so that users can find information acrosspersonal file systems, Outlook or email systems, calendars, and otherdesktop environment.

Web Search

The other spectrum of the search is Web search. There, the story is morelike driving in Boston for the first time. You are not necessarily theexpert of the topics you are looking for and you are learning a newsubject. Sometimes, you search to find new services such as weather,travel, or shopping. With Web search, you are counting on millions ofpeople on the Web to help you and you do not necessarily know or carewho is the real expert or authority. As a result, you sometimes get badadvice or may shop in the wrong places. Web search before Google reliedonly on technologies such as inverted indexes, natural languageprocessing (NLP), and database indexes. They were OK but not as good asit could be if counted the number of links that point at a page. As moresites link to your page, your page becomes more important, simplybecause webmasters behind the sites have gone through the trouble ofadding those extra links to your page. Hence, the birth of page-ranking[tm] and the success of Google's business.

Enterprise Search

The enterprise, however, does not behave as the PC or the Webenvironment. Imagine you are looking for books to learn Javaprogramming—you know your ultimate goal but there are hundreds of booksabout Java, which one should I read: it has to be exactly right. So adiscovery process finds the right reference content, or informationknown by other experts in the company. The ultimate judge of good searchresults for an enterprise extends beyond just yourself. These arbitersof good results could be your peers or the experts that you depend on todo your job.

An example of the problem with enterprise search is shown in FIG. 1,which is a flow diagram showing the state of the art in enterprisesearch. In the example of FIG. 1, Dave is searching for particularinformation and retrieves 2,800 documents. There is no useful resultthat Dave found in the top ten results returned so, Dave calls Sam. Sam,in turn, searches and, finding nothing, e-mails marketing. Mark and Tinain marketing search and find nothing as well. Mark calls Eric, Nancy,and Ganesh and the answer is found in Ganesh's design document. Tinacalls Eric, Nancy, and Ganesh again and everybody is now upset. Clearly,it would have been more useful for Dave if he had found Ganesh's designdocument in his initial search. In fact, the document may have beenthere but among the 2,800 documents located, but it was not possible forDave to identify the most useful document.

Traditional enterprise search technology uses inverted index, NLP, anddatabase index approaches (see FIG. 2). The major problem is that thecurrent engine throws hundreds to thousands of search results per queryback to the user. Anything that looks like Java or programming, is allmixed together for you to see. Much like email spam, search engines spamthe user with numerous, out-of-date, irrelevant, unofficial, siloed,contradictory, and unauthorized results. Users give up quickly andresort to much more expensive ways to get the information includingcalling, emailing, chatting, or worse, starting to recreate, make up, orgive up on the information that already exists.

Enterprise Search Exhibits a Unique Set of Characteristics

By comparing the key issues in enterprise search with that of Web or PCsearch, it can be concluded that enterprise search is unique and indirect contrast to Web search. In fact, what works for Web search doesnot and will not work for enterprise search, and vice versa. Five keyattributes are considered in this regard: search guide, user behavior,freshness and credibility of the content, user homogeneity, and privacyconcerns.

Primary Guide

On the Web, for example, Google's success has depended on page rankingas the primary guide. While page ranking has been effective to providesome sanity in the Web, the same effect will not happen for enterprisecontent search. Firstly, enterprise content lacks the large number oflinks needed to provide the page ranking guiding effect, nor are thereincentives for enterprises to create these links on a sustainable basis.Secondly, the real goal of page ranking is to find the traces of humaneffort to indicate subject authority indirectly because it is next toimpossible to find the real experts in the vast universe of the Web. Forenterprises, you should not need to guess indirectly who might be theexperts, you know who the trusted experts are, you hire them, and theywork day in and day out in the company as specialists in their domainareas. Enterprise search should rely on them as subject authorities forrelevant guidance and ranking.

User Behavior

User behavior is completely different between the enterprises and theWeb. We as individuals on the Web have more faces than we might know. Wecould be men, fathers, sons, husbands, brothers, golfers, travelers,rock musicians, investors, and hundreds of other profiles all at thesame time. When we search on the Web, the search tends to be one-off andall over the place. Also, the keywords we type in tend to be the searchgoals themselves. When we type in “weather,” we are looking for weatherinformation. User feedback on the Web is not reliable because only avery small group of loud users have the time to give feedback andtherefore skews the search results with their non-representative bias(how does this last sentence connects to the rest of the paragraph?Perhaps build a short paragraph that explains the bias in userfeedback).

Enterprise search, however, tends to repeat itself quickly based on theuser's role and the situations he is in. When one sales person islooking for some sales collateral, other sales people responsible forthe same products in the same region are very likely in need of the sameinformation. Equally important is the fact that this person who may have300 roles and profiles in their personal life, has a much smaller numberof work roles, e.g. a half dozen at most. He might be an engineer,working in the Paris office while he is a member of the cross-functionalcultural committee. It is also important to note that the keywords inthe enterprise searches are more like hints, even fishing bait, todocuments a person is looking for. It is thought that eighty percent ofpeople seek information they have seen before. Given the enterprise userpredictability, we can safely rely on self-motivated actions andbehaviors to collect unbiased feedback.

Freshness and Credibility

Web search rewards or ranks older content higher. The longer the contenthas been sitting there, the more likely it will be found because it hastime for others to discover and link to this piece of content.

Enterprises want to behave differently. Fresh content reflects newbusiness situations and, therefore, must be ranked higher so that morepeople see it. By responding to fresh content quickly, business agilityis assured. A piece of content that is one week old may be better thanone that is a year old, except that it is not good at all if today'scontent is available and shows something different than the one week oldcontent. Enterprise search users do not want good enough content, theyrequire the search result to be exactly right.

Homogeneity

The Web or consumer world is very heterogeneous, while an enterprise isthe opposite: homogeneous, or more precisely, segmented homogeneous,meaning that different departments or groups (sales vs. marketing vs.engineering) in a company might be different (segmented), but within agroup, people are very similar or homogeneous in the way they workregardless how different their profiles are.

The implication of this splitting attributes is profound. In a largeheterogeneous world with millions of people involved, statistics is theonly known technique to approach the problem in the effort ofunderstanding what people like, want, etc. Web search relies onstatistics correctly to find not-so-precise information for the users.The enterprise again is different. With small sample populations andhomogeneous groups, statistics do not work. To understand them, you needto know their likes and dislikes. No predictions (what do we mean by‘predictions’?), just awareness.

With this understanding of enterprise characteristics, it is seen thatenterprise search needs to focus on subject authorities, repeatedrole-based work patterns, fresh and official content, and group know-how(a group's collective knowledge and expertise to do a job). Re-examiningtraditional IR-based (information retrieval) search, we realize that itfocuses on the opposite. It relies on the whole content population(crawl and index it) instead of subject authorities, word or linguisticpatterns instead of work patterns, older existing content instead offresh or official content, statistical trends to predict instead ofgroup similarity to know. There is thus a need for techniques that focuson the correct key characteristics of the enterprises.

The problem with enterprise search technology has become acute to manyCIO's and business executives. In the inventor's own limited surveys ofa dozen CIOs and business executives, people ranked the enterprisesearch priority problem as a 9-10 out of 10. The challenge oftraditional full-text engines is poor relevancy. They are good foreverything (all content) and good for nothing (irrelevant results) atthe same time. The NLP technology achieves better relevancy by focusingon one application and one domain where human language becomes moredeterministic. The problem with the NLP is that the solution is placedin a silo and good only within that specific application, whileenterprises are operating on hundreds to thousands of applications. Itis not possible for employees to log on to these many systems one by oneto look for information. Both classes of solutions also suffer from theinability to adapt to changes once deployed. Taxonomies and structureschange quickly over time in enterprises.

Current search software also suffers from traditional enterprise modelwith inherited expensive product architecture, design and marketing andsales model. A typical enterprise search deployment costs $500K toseveral millions after considering software licenses, services,training, and other related costs.

It would therefore be advantageous to transform how enterprise searchtechnologies are bought and deployed with an improvement on cost andquality of search.

SUMMARY OF THE INVENTION

The invention addresses the above limitations of state of the artenterprise search by leveraging what should be depended on forenterprise search: one's peers and experts in and out of the company.The invention provides systems that identify, extract, analyze, and usethe expertise ranking to produce personalized, precise search results tothe user so that they do not have to call, email, etc.

The inventors have discovered a set of unique approaches to enterprisesearch that is different from all existing IR (information retrieval)based solutions, such as Verity, Autonomy, FAST, Endeca, and GoogleAppliance. The inventors carefully analyzed the characteristics ofenterprises in contrast to the Web search environment, and applied a setof methodologies in related disciplines from technology development,academic research, and social behavior. The invention provides atechnique that can work standalone or embed itself in other applicationsvia a plug-n-play interface with minimum effort. The result is a hugeimprovement in search usefulness, relevancy, search federation acrossapplications, and cost savings. The preferred embodiment of theinvention also leverages traditional search technologies.

The invention provides relevant information discovery by taking acompletely opposite approach to that of traditional search theories andtechnologies. As discussed above, traditional content search technologyand products use content as the basis for guiding searches. It employstechniques such as information retrieval (IR) algorithms, naturallanguage processing (NLP) techniques and rules, product or structuraltaxonomy, or page ranking by link count. Traditional data search relieson building database indexes on key words or numbers in database rows orcolumns. It crawls and indexes the content and data, generates invertedfull-text indexes or database indexes with word tokens or phrases,potentially assisted by taxonomy and paging ranking for improving searchresults. The search results using traditional search technology arepoor, with large amounts of low-relevancy hits. For many businessprocesses, when a search fails, users have to resort to alternative,expensive ways of acquiring information that either take a significantamount of time for the user, or worse yet, involves others to help findthe information (see FIG. 1).

Instead of using content as the starting point for informationdiscovery, the invention provides a system that starts with the peoplein and around enterprises. After all, enterprises are made ofspecialists and experts possessing expertise and know-how. They conductwork and repeat their work patterns frequently on a role-by-role basis.The system detects and captures the expertise and work patterns storedin people's brains and exhibited in their daily behavior, and creates abehavioral based knowledge index. The knowledge index is then, in turn,used to produce expert-guided, personalized information. This process istransparent to the experts themselves, and therefore efficient andextremely economical to employ.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flow diagram showing the current state of the art inenterprise search;

FIG. 2 is a flow diagram showing traditional IR-based search models;

FIG. 3 is a block schematic diagram showing system architectureaccording to the invention;

FIG. 4 is a flow diagram showing the capture of behavioral relevancy byan embedded application according to the invention;

FIG. 5 is a screen shot showing an inline user interface according tothe invention;

FIG. 6 is a screen shot showing an inline user interface rendered usingJava Script tags according to the invention;

FIG. 7 is a screen shot showing a popup user interface according to theinvention;

FIG. 8 is a block schematic diagram showing expert-guided personalizedsearch across applications according to the invention;

FIG. 9 is a screen shot showing a user library according to theinvention;

FIG. 10 is a second screen shot showing a user library according to theinvention;

FIG. 11 is a third screen shot showing a user library according to theinvention;

FIG. 12 is a flow diagram showing a document recommendation according tothe invention; and

FIG. 13 is a flow diagram showing an augmented search according to theinvention.

DETAILED DESCRIPTION OF THE INVENTION

Using Expertise and Behavior for Information Discovery

The invention comprises a set of complementary techniques thatdramatically improve enterprise search and navigation results. The coreof the invention is an expertise or knowledge index, also referred to asan expertise repository, that makes observations of website and webapplication visitors. The expertise-index is designed to focus on thefour key discoveries of enterprise search: Subject Authority, WorkPatterns, Content Freshness, and Group Know-how. The invention producesrelevant, timely, cross-application, expertise-based search results. Incontrast, traditional Information Retrieval technologies such asinverted index, NLP, or taxonomy tackle the same problem with anopposite set of attributes than what the enterprise needs: ContentPopulation, Word Patterns, Content Existence, and Statistical Trends.

A further embodiment of the invention makes the novel technology workwithin existing an enterprise application and repository environmenttransparently so that no user training or adoption of new interfaces isrequired. It also supports all legacy full-text or NLP searchtechnologies, such as Verity, Autonomy, Endeca, and the GoogleAppliance. In fact, it works on top of those technologies and uses theirbase result as a foundation for refinement.

A third embodiment of the invention comes from leveraging open sourcetechnology, such as Lucene, for building a scalable network query enginethat binds all dimensions of the information source and indexes into oneset of meaningful results.

Embeds in Application UI to Capture Behavioral Relevancy

The invention embeds itself in any existing Web applications such aswww, CRM, ERP, and portals etc. via a simple change to the searchresults interface. For non-Web applications, similar work can be done byinserting SOA (service-oriented architecture) stub code so that searchtraffic can be inspected and re-ranked by an expertise index. Further,any web page, not just search results pages, can be configured with theinvention to provide active guidance to the user without requiring theuser to enter a query.

Reliance on Self-Interest

The invention does not require users to explicitly vote, providefeedback, or utilize other mechanisms that commonly result incollaborative filtering. It relies on people doing their normal jobsselfishly and leaving a trail of evidence of what they need and preferto get their job done. The reliance on selfishness is fundamentallydifferent and far more reliable guidance than the traditionalcollaborative filtering, where users are instructed to vote for otherpeople. When users are asked to give feedback, most people do not do itbecause they lack time or it is not a priority. When forced, peoplecheck boxes quickly without thinking and, therefore, mislead people whouse the data. There is a small group of people who do like to fill outsurveys and give feedback, but often times they are the vocal, critical,and least representative samples of the user population. Both Amazon andeBay have negative experiences in using traditional collaborativefiltering techniques to accomplish ranking by similarity.

Implicit Relevance Actions

The invention allows application-by-application configuration of useraction tracking. Implicit action buttons (discussed below) are embeddedas part of the search results to capture critical cues of user intentionand preferences as the users do their job. For example, a common portalmay give users “view,” “download,” “print,” and “email” buttons as theactions to reflect their intention when discovering relevant content.“View” might be a weaker indication, while the others are strongindications of preference. The invention develops additional implicitobservations that predict visitor intentions with strong confidence.These observations include the ability to detect think time, virtualbookmarks, virtual print and virtual email. In all cases, visitors havenot performed a bookmark, print, or email against the content, but theykeep the content up on the computer screen for a long time, i.e longenough to use the content as a reference for work. These observationsare cross checked among peers and experts before they truly becomeuseful for the community.

Explicit Relevance Actions

At least two additional explicit buttons can be added to track clearcues of user behavior. “Save to library” indicates a strong, explicitendorsement of content given the query, while “remove” or “demote”indicates strong dislike of the content given a query and a role. Thelibrary is virtual and does not physically live on a browser or even onthe PC. It is the main user behavior tracking object or a journal.Again, explicit relevance ranks higher than implicit relevance, but bothare managed under one per-user library object.

Search Spam Control

People are familiar with email spam. Current search exhibit similarbehaviors to spam, with thousands, if not tens of thousands of resultsreturned in response to a simple query, and many of the results aretotally irrelevant to what the users are looking for. Through the use ofthe “remove” action after considering the role an employee is playing,the inventive system can identify and demote results that are lessrelevant or irrelevant to a group of users of the same role, in a mannerthat is analogous to the spam email reporting scheme. For example, ifthree engineers remove their interest in a document, other similarengineers should not see this document highly ranked, while salesemployees may still give this document a high ranking.

Expert-Guided, Personalized Search Across Applications

Consider the invention in the context of the entire enterprise. Althoughuser actions are done through business application interfaces on theirPCs, a behavioral journal called “my library” is stored and maintainedin the system server. Neither the PC nor the application involved needsto be concerned about the journal.

My Library/Behavioral Journal

This is a per user object. It is generally invisible to the users unlesspower users or applications want to use them directly. The data in thelibrary can be mined or learned before the system goes live. Itcontinues to improve itself, adjust, and adapt to the real businessusage of the content and their queries. The library stores user profilesand attributes, all queries, relevant content URIs, one or more indexesfor all relevant content and data, query caching, content and datacaching, access time, personalized ranking formula, proximity hashing,and a loading ratio control for the privacy policy considered.

A desktop version of My Library can be added to provide content caching,content push/update/alert, and disconnected content access.

Domain Expertise Networks

One aspect of the invention concerns examining multiple personallibraries or behavioral journals. When enterprises start to analyze manypeople's journals from peer and expert dimensions, great insights oninformation consumption and employee productivity emerge.

Peers are defined herein as a group of users with common interests, suchas products, topic sets, events, job roles, locations etc.

Experts are defined herein as a group of visitors with a differentknowledge and skill sets than the person querying or browsing, but theperson querying and browsing depends on the experts to do his jobeffectively.

For example, an engineer has a peer group of other engineers, and alsohas an expert group made up of product managers, sales people, somecustomers, HR staff, and office assistants. Peers and Experts can changewhen context is changed. An engineer, John, may play roles beyond theorganization he is in. He could be a cross-functional committee member,and physically work in London office. So John has three contexts that heis part of. This context is referred to herein as a Domain ExpertiseNetwork or DEN. An employee may belong to several DENs. Various types ofDENs are discussed in greater detail below.

Architecture

Expertise Index System also referred to as an Expertise and BehavioralRepository: This element is key to the invention. It is a server basedsystem with a service-oriented architecture (SOA) using Web services,XML, J2EE, and other foundational technologies.

The following components are key parts of the invention:

Behavioral Instrumentation: Also referred to as a Work Monitor, thiselement is responsible for implanting and recording user behavior onvarious business applications. The application search form is one ofmany observation posts that the invention implements. Browser andapplication navigation, file upload, Web plug-in, page-tags, emailserver and client integration, content management, document management,records management, and collaboration systems are all common places forinstrumentation. The invention also goes back in time, and parses commonlog files, such as Web server logs, query logs, directory files, e.g.LDAP, to build and extract historical or base level expertise.

Real-time Behavioral Journals: This element is a per user objectdescribed above.

Domain Expertise Networks: This element is the work relationship object,described above, that connects personal, peer, and expert associations,and that records repeated role-based enterprise work patterns.

Non-Uniformed Network Index (NUNI): This element provides the mostrelevant, timely and authoritative search results. This is discussed indetail below.

Contextual Mapping and Dynamic Navigation: With the help of personal,peer, and expert journals, the NUNI index can not only produce goodsearch results, but also provide additional contextual information thatthe users are not directly asking via their search queries or keywords.The contextual results can be presented back to the users in a searchresult sidebar, or as part of personalized, dynamic navigation. Dynamicnavigation is discussed in greater detail below.

Productivity Reports and Solutions: This element generates variousreports based on the behavioral journals and NUNI index.

In summary, the Expertise Index System focuses on enterprise SubjectAuthorities, Work Patterns, Content Freshness, and Group Know-how todeliver expert-guided, personalized information.

Technical Overview

FIG. 3 is a block schematic diagram showing the system architecture of apreferred embodiment of the invention. A more detailed discussion ofvarious aspects of the architecture is provided below. The architectureconsists of a server farm 20, a customer enterprise 22, and a userbrowser 21. The user browser is instrumented with an extension 23 andaccesses both customer servers 25 at the customer enterprise 22 and theserver farm 20 via a load balancer 27. Communication with the serverfarm is currently effected using the HTTPS protocol. User access to thecustomer server is in accordance with the enterprise protocols. Thebrowser extension 23 is discussed in greater detail below.

Of note in connection with the invention is the provision of a failsafe.The extension 23, as well as the enterprise extension 24, areconstructed such that, if the server farm 20 does not respond in asuccessful fashion, the extension is shut down and the enterprise andbrowser interact in a normal manner. The features of the invention areonly provided in the event that the server is active and performing itsoperations correctly. Therefore, failure of the server does not in anyway impair operation of the enterprise for users of the enterprise.

As discussed above, an extension 24 is also provided for the enterprisewhich communicates with the load balancer 27 at the server farm 20 viathe HTTPS protocol.

The enterprise also includes a helper 26 which communicates with theserver farm via an agency 31 using the HTTPS protocol. The agencyretrieves log information from the enterprise and provides it to loganalyzers 28, which produce a result that is presented to the usagerepository 29. Information is exchanged between the affinity engine 32and the browser and enterprise via various dispatchers 30. The browseritself provides observations to the server and receives displays inresponse to search queries therefrom. These observation and displays arediscussed in greater detail below.

A key feature of the invention is the affinity engine 32 which comprisesa plurality of processors 33/34 and a configuration administrationfacility 35. During operation of the invention, a form of information,also referred to as wisdom, is collected in a wisdom database 36. Theoperation of the affinity engine is discussed in greater detail below.

The inventive system is typically installed as an enhancement to anexisting search system based on conventional engines provided byvendors, such as Verity, Autonomy, Google, etc. This content and datasearch system based on conventional technology is referred to as theexisting search mechanism.

The inventive system is implemented as a wrapper for an existing searchmechanism. When a user issues a search query, the query is handledinitially by the system. The system, in turn, typically forwards thequery to the existing search mechanism. It may also perform one or moresearches or related operations against its own internal indexes anddatabases. Once the results from the various searches have beenobtained, they are merged together into a single set of results. Theactual presentation of these results is at the discretion of thecustomer, who may either take the raw results data from the system andpresent them using a JSP, CGI, or similar mechanism, or else use thedefault search results page provided with the system, possiblycustomized using cascading style sheets or other similar techniques.

Each document in the results is generally presented along with a varietyof possible actions for the user to take on the document. The availableactions are site-configurable, and can include, for example, “think”,“view,” “download,” “email,” or “print.” The system is informed when auser selects one of these actions for a particular document. That dataare then used to infer the relevance of a particular document withrespect to the query that yielded it. Thus, if a user selects the “view”action for a document, the system might infer that the document hascertain actual value to the user for that query, while if the userselects a more permanent action such as “print” or “download,” thesystem might infer that the document is highly relevant to the user. Thesystem can detect virtual print or download to give an accurateapproximation as if a physical print, download, or bookmark hashappened. The techniques rely on detecting activities of users on thebrowser for a certain amount of time, e.g. over one minutes, wheredocuments remain open for a long time, i.e. long dwell. On the otherhand, if a user does not perform any action at all against the resultsfrom a query, the system might infer that the results were irrelevant tothe user. This data are retained and used to influence the results offuture queries by the user and to generate quality metrics.

Libraries

The system maintains a library of content reference and/or use for eachuser. The library is also called the behavioral journal. This library issimilar in some sense to bookmarks in a Web browser, though it is notnecessarily visible to the user. Indeed, the user may not even be awareof its presence. Depending on how the system has been configured, adocument name and its location may be added to a user's libraryautomatically when certain actions for a document are selected from thesearch results. A document could also be added to a user's libraryexplicitly with an optional “add to library” action from the searchresults. The presence of a document reference in a user's librarygenerally indicates that the document is of particular interest to theuser. Thus, if the results of a query produce a document that alsoappears in the user's library, its ranking is typically improved.

In some configurations, it is possible to add a document to a user'slibrary directly, without first encountering it in search results. Sucha document need not be indexed by, or even accessible to, the existingsearch mechanism. However, because it is present in the user's library,it can still be merged into the final search results if it matches aquery, and it is therefore available in the results produced by thesystem. Content discovered in this manner is typically quite valuableand so is usually given particular preference in the result rankings.

Relations

People in businesses relate to each other in a number of different ways.For example, there are relationships between peers in a group, betweensuperiors and subordinates, or between subject matter experts andseekers. When these different kinds of relationships are modeled andobserved, they reveal insights that can be used to influence and refinesearch results. For example, if several members of a group of peers allfind a particular document to be helpful, then there is good chance thatother members of that same group would find the document helpful as wellbecause members of a peer group typically have similar interests.Similarly, if someone is seeking information about a particular subject,then documents that a known expert in that subject found useful wouldprobably be valuable to the seeker as well.

The system maintains one or more named relations for each user torepresent these kinds of relationships between one user (the subject)and other users (the related users) in the system. A relation isformally the set of users that have a particular relationship with thesubject. A relationship can be two-way or one-way. A two-wayrelationship applies equally in both directions between the subject andthe related user. Thus, if user A has a two-way relationship with userB, then user B has the same kind of relationship with user A. An exampleof this might be a peer relationship, which could describe two users whoare in the same organizational department or who have similar jobdescriptions: if user A is a peer of user B, then it is also the casethat user B is a peer of user A. On the other hand, a one-wayrelationship is directed: if user A has a one-way relationship with userB, it is not necessarily true that user B has that same kind ofrelationship with user A. An example of this might be asuperior-subordinate relationship: if user A is a subordinate of user B,then it is not the case that user B is a subordinate of user A.

Because related users are users of the system, they have libraries oftheir own. Depending on the configuration, the system can search thelibraries of some or all related users as part of a query and merge anyhits into the results. The degree to which results from a related user'slibrary biases the baseline results can be configured both at therelationship level, e.g. experts have a larger bias than peers, and alsoat the user level, e.g. some peers may exert more influence than others.

In the default configuration, the system maintains two differentrelations for each user:

Peers: This is a two-way relationship intended to represent users withcommon interests, job roles, locations and other factors. People canbelong to multiple peer groups based on different contexts. The systemdevelops the peer groups through learning. Peer group change and adaptaccording to community and business changes.

Experts:, This relationship represents skill sets or knowledge a personpossesses. The system detects experts by examining the community andindividuals who have the ability to discover and collect the most anduseful documents having the most impact. Experts are relative. An experttoday may become less so if the person stops to be the connection to themost useful content.

Although this typical configuration only provides a single peersrelation and a single experts relation for each user, an advancedconfiguration might supply two or more of each, with each peer andexpert pair called a Domain Expertise Network (DEN). Multiple peersrelations or DENs allow a user to identify several different peer groupsthat are each relevant at different times, e.g. a departmental group forday-to-day operations, a special interest group representing a committeemembership, etc. Multiple experts groups allow a user to have severaldifferent sets of experts focused on different subject areas.

Monitoring User Activity

FIG. 4 is a flow diagram showing the capture of behavioral relevancy byan embedded application. In FIG. 4, an information seeker is using abusiness application. In doing a search, such as “sales preso on bonds,”the server performs various data mining activities and produces a resultfor the information seeker. In the process of doing so, the inventionmakes observations of implicit relevance actions, such as “view,”“download,” “print,” and “e-mail.” The server also makes observationswith regard to explicit relevance actions, such as “save to library,”and actions similar to spam control, such as “remove.” These items arediscussed in greater detail below. The observations made by the systemare used to determine the value of a particular document to a searcher.The system accumulates information about the value of the document andthen develops a usefulness measure for the document, as discussed ingreater detail below.

FIG. 5 is a screen shot showing an inline user interface according tothe invention. Because the tags used in the system are configurable andcustomizable, the user interface can be made to blend into an existingWeb site for a particular enterprise. The example given in FIG. 5 of apublic Web site.

FIG. 6 is a screen shot showing an inline user interface (UI) renderedusing JavaScript tags according to the invention. This particularexample shows the “most popular” tag, which gives a list of the mostpopular documents to the end user. The UI is rendered using JavaScripttags. Other tags, such as “next step,” “similar documents,” and“preferred” are rendered in a similar fashion.

FIG. 7 is a screen shot showing a pop-up user interface according to theinvention. As with the inline user interface, this interface is renderedusing JavaScript tags. This particular example shows a “next step.” Thistag fades in and, when closed, out to enhance the user experience. Aswith the inline tags, the pop-up dialogue is also configurable to blendinto any existing Web page's style.

Action Monitoring/Observations

The system has an active interest in knowing which documents users findhelpful or relevant. However, users cannot generally be relied upon toindicate explicitly to the system when a particular document isconsidered helpful or relevant. Instead, the system has to infer thisinformation from actions users would take anyway on a document, with orwithout the system.

One way to do this is to present to the user one or more convenientbuttons or links for typical actions with each document in the searchresults. Because these actions are available with a single mouse click,as opposed to the multiple clicks that are typically required to performmost actions using normal browser controls, users tend to use themrather than the standard browser controls for performing these actions.Furthermore, because these buttons or links are under the control of thesystem, the system is able to take note of the actions a user takes withrespect to a document. Thus, users are given a convenient mechanism forperforming actions they would perform anyway on documents in a set ofsearch results, and the system is able to monitor these actions.

In practical terms, intercepting these user actions is straightforward.As is normally the case with HTML, each button or link representing anaction has a URL associated with it. Normally, such a URL would referdirectly to the associated document. However, with the system these URLsinstead refer to a CGI, servlet, or similar mechanism associated withthe system. The URL contains information about the user, the document,and the action the user wants to perform. The system logs the action andrelated information, and then redirects the request to either theoriginal document, in the case of simple “view” type actions, or someother kind of Web resource to complete the requested action.

Most content search systems make the title of a document an active linkto the document when presenting search results. The system uses thisstandard convention as well, except that the active link is treated as a“view” action and monitored in the same manner as the other actionsdescribed above.

An optional “add to library” action is available for documents as well.As the name implies, this action adds the document to the user'slibrary. This is a way for users to inform the system explicitly that adocument is particularly useful. A user's primary motivation for usingthis action is to ensure that the document is considered favorably infuture queries because documents in a user's library are generally givenimproved rankings.

When a user elects to take responsibility for presenting search resultsmanually, URLs for the configured actions are provided along with theother usual data for each document in the results. It is the customer'sresponsibility to ensure that these URLs are used for the variousactions users might take on the result documents, or else the value ofthe system is diminished.

General Implicit Observations of Search and Navigation

The system uses more generic facility to observe user behaviors againstall content during search and navigation. The observations are madeimplicitly without user participation other than their doing theirnormal browsing and searching. Observations are consolidated from eithersearch and navigation and then used to improve future search andnavigation.

Query Monitoring

The system also benefits from knowing the original query that yielded adocument on which a user takes an action. For example, if the systemnotices that a user issues the same query later, or if it noticesseveral different users making the same or similar queries, it canincrease the ranking of documents in the new query that were foundinteresting in the original query. However, because query strings can berather cumbersome, it is not always practical to include them in theaction URLs. Instead, the system maintains a database of query stringsand issues a unique ID for each. This unique ID can then be includedwith the action URLs presented in the search results. When a user takesan action on a particular result document, the system can determine thequery that produced that particular document by looking up the query ID.

Blended Search

The system uses blended search to enhance search results. In a blendedsearch, a single query is passed to two or more separate searchprocessors, each of which produces a set of zero or more documents,referred to as the result set, that match that query in some fashion.Depending on the configuration and circumstances, the same document mayshow up in one or more of the result sets from these various searches.Once all of the search processors have completed the requested query,their result sets are merged together into a single result set. Rankingsare assigned to individual documents in the merged result set using aconfigurable formula that takes into account such factors as the numberand/or type of search processors that produced the document and thedocument's ranking within each of those individual result sets.

The two distinct search processors need not be distinct softwareentities. For example, the same search engine running against twodifferent indexes and/or with different configuration parameters couldconstitute two distinct search processors. More important is that twodistinct search processors should typically yield different results forthe same query. One might consider that each search processor offers adifferent point of view for a query.

Each search processor can be assigned a weight that determines thedegree to which it influences the rankings in the merged search results.This weight can be either a static constant, or a dynamically computedvalue that varies according to the query, results, or othercircumstances.

Search processors can, but do not necessarily, run independently of eachother. Some search processors can be configured to take the result setof a different search processor as input and manipulate it in some wayto produce its own result set. This kind of search processor is referredto as a filter. Filters are useful for such tasks as narrowing theresults from a different search processor, e.g. removing documents thatare too large, too old, etc., or modifying them in some way, e.g.computing summaries or titles from document contents, adding annotationsfrom revision logs, manipulating the ranking score, etc. A searchprocessor that does not filter the output of another search processor isreferred to as an independent search processor. An ordered sequence ofsearch processors in which the first is an independent search processorand the second and subsequent search processors acts as filters for thesearch processors preceding them is referred to as a pipeline. Theindividual search processors that make up a pipeline are also referredto as stages.

The result set of a blended search is formed by merging the outputresult sets of one or more pipelines. As a rule, each pipeline producesa score for each document in its result set that is used for ranking thedocument's relevance. When the results of two or more independent searchprocessors are blended, these scores are normalized to the same range,then multiplied by a scaling factor. If the same document appears inmore than one pipeline's result set, the scores from each result set areadded together to form a single score in the blended result. Theseaccumulated scores determine the final rankings of the documents in theblended results, with the highest scores being given the best rankings.

As a practical matter, separate pipelines can be run in parallel forefficiency. On a conceptual level, the various stages of a singlepipeline are run serially, though in actual practice some parallelismcan still be achieved when stages are able to produce portions of theirresult sets incrementally. The composition of the pipelines that areused in a blended search and the manner in which they are run, e.g.serial vs. parallel, is configured by the administrator and/ormanipulated dynamically by the end user.

In a typical configuration, the existing search mechanism that is beingwrapped by the system is referred to as the baseline processor. Anyother search processors are referred to as ancillary processors. Abaseline processor is normally built on top of conventional searchtechnologies and is therefore capable of standing alone as an adequate,though sub-optimal, document search mechanism. Amongst other things,this implies it should have access to the majority of public documentsin an enterprise, have a query processor capable of handling typicalrequests from most business users, and that it not act as a filter stagein a pipeline. Ancillary processors, on the other hand, have fewer suchrequirements: they may have access to only a handful of documents, theymay or may not use a conventional search engine to accomplish theirgoals, and they may in fact participate as a filter stage in a pipeline.

Note that the system can in fact be configured with two or more baselinesearch processors. This is sometimes referred to as federated search, inwhich the results of otherwise independent search engines are merged.Though this is not necessarily a goal of the system, it is a beneficialspecial case of its blended search technology.

FIG. 8 is a block schematic diagram showing expert-guided personalizedsearch across applications according to the invention. In FIG. 8, theserver is shown including information about the user's library, “MyLib.” The user's browser 21 is shown having a “My Lib” view. The sourceof this view includes searching of a business application, Web searches,and other business application information. This creates a networkeffect so that other applications can use the server as well. The user'slibrary is a behavioral journal. It can be embedded in otherapplications and is, therefore, not just a new user interface orapplication. The contents are created by user search and discovery andare generally invisible to the user. The analytics of the system allowthe improvement of quality and provides bridge silos. As discussedherein, there is a form of spam control implicit in operation of theinvention. The system provides dynamic personal navigation support. Aproximity hash, loading ratio, and privacy C policy are alsoimplemented. The invention operates in the form of a browser and desktopplug-in and includes content update and caching. The informationaccessed in connection with the invention is pursuant to a domainexpertise network, discussed in greater detail elsewhere herein, thatconsists of individual information, peer information, expertinformation, and community information.

FIG. 9 is a screen shot showing a user library according to theinvention;

FIG. 10 is a second screen shot showing a user library according to theinvention; and FIG. 11 is a third screen shot showing a user libraryaccording to the invention.

Sample Search Processors

The system can be realized with different search processors provided inconnection with the affinity engine (FIG. 3), that can be combined indifferent ways to accomplish different goals. The following discussiondescribes several of the more common search processors that areavailable.

Lucene Baseline Search

This search processor is an independent baseline processor thatgenerates its results by issuing a query to an existing Lucene index(see http://Lucene.Apache.org). The result set that it produces includesa content locator and a relevance score that is a floating-point numberin the range of 0.0 to 1.0.

Library Search

This search processor is an independent ancillary processor thatsearches a particular user's library for documents that match aspecified query. In a typical implementation, a Lucene index ismaintained for each user's library, so this search processor isessentially a special case of the Lucene baseline search processorrunning with a different scaling factor against a different index.

My Library (My Lib)

This special case of the library search processor runs against thelibrary of the user that has invoked the original query. It normallyruns with a relatively large scaling factor. Thus, documents in whichthe user has previously shown interest and which match the current querytend to receive elevated rankings.

Relation Search

This search processor is an independent ancillary processor thatsearches the libraries of the related users in a given relation. It isconceptually similar to invoking the library search processor for eachof the related users then merging the results. In practice, this can beoptimized in a number of different ways, for example by performing eachlibrary search in parallel, or by maintaining a separate merged indexfor the entire relation.

My Peers

This search processor is a case of the relation search processor thathas been specialized for one of a subject's peer relations. If the userhas more than one such relation, the specific relation to be used for agiven search can be determined in a number of different ways.

For example:

-   -   It can be set by an explicit action on the part of the user,        e.g. the user might indicate that work is currently being done        in the context of a particular peer group;    -   It can be set implicitly by the current search context, e.g. the        actual search form used to launch the query might select a        specific peer group;    -   It can be computed, e.g. by analyzing the query itself.

The theory behind this search processor is that a user's peers tend tohave similar interests to the user, so if a document was particularlyinteresting to a peer, i.e. the document is in the peer's library, thenit probably is interesting to the user as well. This search processorgenerally runs with a relatively high scaling factor, thus elevating therankings of documents that both match the query and reside in a peer'slibrary.

Transitive Relation Search

Many, but not all, one-way relationships are transitive: if user A has aparticular one-way relationship with user B, and user B has a similarone-way relationship with user C, then if the relationship is transitiveit can be inferred that user A has this same one-way relationship withuser C. If a given relation represents a transitive one-wayrelationship, then the transitive closure of that relation is the unionof the members of the original relation with the members of the samerelation for each of those related users. In a full closure, thisprocess is continued recursively for each of the related users and eachof their related users, etc. until the full tree of transitiverelationships has been computed. In a partial closure, the recursion islimited to a particular depth.

The transitive relation search processor is an independent ancillaryprocessor that searches the libraries of all users that belong to a fullor partial closure of a specified one-way relation. A single recursiondepth can be specified for the entire relation, or a separate recursiondepth can be specified for each member of the starting relation. Oncethe closure itself has been computed, the transitive relation searchprocessor is very similar to the normal relation search processor,conceptually performing a library search on each user in the closure andmerging the results. For this reason it can be optimized in the samemanner as the regular relation search.

My Experts

This search processor is a special case of the transitive relationsearch processor that has been specialized for one of a subject's expertrelations. If the user has more than one such relation, the specificrelation to be used for a given search can be determined in a number ofdifferent ways, as outlined for the My Peers search processor.

The theory behind this search processor is that if a user can identifyexperts in a particular subject, then documents that those experts findinteresting, i.e. that appear in the experts' libraries, are presumablyof interest to the user as well when conducting a search in thatsubject. Furthermore, it is assumed that expert relationships aretransitive. Thus, if user A considers user B to be an expert on sometopic, and user B considers user C to be an expert on the same topic,then user A would consider user C to be an expert on that topic as well,even though user A does not necessarily know user C.

As with the My Peers processor, this search processor runs with a highscaling factor thus causing content selected by experts to be givenelevated rankings.

Freshness

One important contrast between content searches in an enterprisecompared to more general Web search is the importance of freshness orrecency in the results. On the Web, somewhat older data are generallyconsidered more valuable because they have had a chance to be evaluatedand vetted by users around the world. In an enterprise, however, theopposite is true: most users have already seen the older data, so when asearch is performed newer data are usually more useful.

The freshness search processor is a simple ancillary filter processorthat captures this difference by increasing the scores of more recentdocuments and decreasing the scores of older documents. The degree towhich a document's score is changed varies according to its age. Thusvery recent documents might have their scores increased more than lessrecent documents, and very old documents might have their scoresdecreased more than middle-aged documents. The thresholds and ranges forthe various types of scaling are all configurable, making it possible,for example, to set up a filter that only penalizes old documentswithout enhancing new documents, or contrarily, to penalize newdocuments and enhance old ones.

Explicit Bias

Some documents are the canonical correct answer to certain queries. Forexample, in organizations that must pay special attention to regulatorymatters, e.g. HIPPA, SOX, etc., a query related to a particularprocedure is ideally answered with the most current, officialdescription of that procedure, possibly to the exclusion of all otherdocuments.

The explicit bias search processor is an ancillary processor thatrecognizes certain queries or query keywords and injects a fixed set ofdocuments in the results for those queries, each with a fixed score,usually a very high one. This is generally done without a formal searchindex. Typically, it is configured with a simple table that mapskeywords to documents. It can be configured as either an independentprocessor or a filter. When it is configured as a filter, it can furtherbe configured to either replace or supplant the input results. When theexplicit bias search filter does not find a matching keyword, it leavesthe input results unmodified.

Popularity

Some search topics tend to recur regularly in any given enterprise,typically with a small number of documents in the results towards whicheveryone gravitates. The system can detect these popular results bynoticing when the same query is issued multiple times and then watchingwhich documents are acted upon most frequently in response to thesequeries.

The popularity search processor is an ancillary filter processor thatputs this knowledge to use. It detects popular queries and thenincreases the ranking of documents in the results that have historicallybeen selected by previous users making the same query. In practicalterms, it is similar to the explicit bias processor, except that thetable of keywords to documents is generated automatically by the systemfrom data obtained by analyzing the query and action logs.

Quality Metrics

Because the system watches both queries and the actions taken on theresults of queries, it can monitor the quality of its resultsdynamically. This is then used for such purposes as return-on-investment(ROI) reports or feedback on site design.

A simple form of feedback on search quality can be found be comparingthe query logs to the action logs. If a user query produces nocorresponding actions, or perhaps only yields actions on poorly rankeddocuments, then the system can infer that the query produced poorresults. On the other hand, a query that yields several differentactions, particularly to highly ranked documents, might be consideredgood.

Another dimension for quality feedback is to compare actions ondocuments that would have been found by the baseline search processor tothose that would have been found only by the system, or perhaps todocuments that were found more easily because of the system. The systemcan accomplish this by taking note of which search processorscontributed significantly to a document's relevance score when an actionis actually taken on that document. If the only significant contributorto a document's score was the baseline search processor, then the systemcan infer that it did not add any particular value to that result. Onthe other hand, if one or more of the search processors contributedsignificantly to the document's score, then the system can infer that itdid add value to the result.

By combining these two metrics, the system can dynamically produceinteresting and valuable ROI reports. For example, one report might beto compare the ratio of good-to-poor search results for queries thatwere enhanced by the system to the same ratio for queries that were notenhanced by the system. If a dollar cost is assigned to poor queries,then the difference in the cost of poor search results rendered by theoriginal search system and those rendered by the system can be computed.Another report might concentrate on the amount of time the system savesits users. For example, a document that was found only by a searchprocessor and not the baseline search processor might be assumed to savethe user two hours of manual research, while a document that was pushedfrom a low rank to a high rank by search processors might be assumed tosave the user 30 minutes of research. If a cost per hour is assigned tothe user's time, then a cost savings for using the system can becomputed.

Non-Core Technology

Because the system is designed to wrap an existing document searchmechanism, it necessarily employs a number of technologies that are notintrinsically its own. The following discussion describes these types ofnon-core technology used by the system. Those skilled in the art willappreciate that the following is only an example of a presentlypreferred embodiment of the invention, and the other technologies may bechosen to implement the invention.

Language

The bulk of the system is implemented using version 1.5 of the Javalanguage, and all classes are compiled using the Java compiler suppliedby Sun Microsystems in 1.5.04 of their Java Software Development Kit. Itis presumed to run correctly in any JVM supporting version 1.5 of theJava language. If customers do not provide a JVM of their own, version1.5.04 of the Sun JVM are used by default.

Application Server

Much of the core functionality of the system is implemented using Javaservlets and Java Server Pages (JSP). The current implementation iswritten to version 2.4 of the Java Servlet specification and version 2.0of the JSP specification. It should, in principle, run in anyapplication server supporting those specifications. If customers do notprovide an application server of their own, version 5.0.28 of the ApacheTomcat application server is used by default.

Search Engine

The system uses version 1.4.1 of the Lucene search engine to manage userlibraries. The current implementation includes support for Luceneversion 1.4.1. If customers do not provide a baseline search engine oftheir own, a basic implementation using Lucene version 1.4.1 isprovided.

Web Server

Any conventional Web server can be used with the system to serve regularcontent. The reference implementation of the system uses Apache 2.0.52.

Tool for Virtual Configuration of Web Applications

A tool that configures a target web application with new capabilities,such that the new capabilities can be demonstrated live within the Webapplication, though the Web application has not been modified in anyway.

Method for Virtual Proof of Concept

An automated process is provided that enables the evaluation of a set ofsoftware capabilities within existing Web applications by guiding theevaluator through a series of steps and automatically provisioning thenecessary infrastructure to support the evaluation. The process isvirtual in that it requires no changes to the target Web application andno installation of software.

One aspect of the invention concerns a virtual Web sales tool. In thisembodiment, the invention comprises a virtual environment that isimplemented using proxy technology. The system is used by a prospectivecustomer to access a system Web site. This allows the prospectivecustomer to see the “before they know” and “after they know” impact ofthe system against the prospective customer's live application. Avirtual environment is created that mimics the prospective customer'slive application, without copying the live application's content. Thesystem nonetheless performs interception and augmentation in this proxyenvironment without physically possessing any content or interferingwith the structure of the live application. Thus, when a prospectivecustomer comes into this virtual environment, they feel as though theyare actually in their live application. One benefit of this embodimentis that the invention may be used to do instrumentation without havingto go physically into a customer's application environment, get logging,or the customer's IT department involved. Thus, the customer does notreally know there is a change, but can see the impact.

In this embodiment, it is possible to go through a process that requiresno installation of software in the traditional sense and that allows acustomer coming to the Website to have the same kind of experience thatthey would traditionally have with a traditional software provider,except the invention allows one to do it all online. This approach isvirtual in the sense that there is nothing the has to do except interactwith a Web browser to take advantage of the service. This embodimentprovides a virtual proof of concept (POC) that automates the salesprocess for the system. The intent is to capture interest through theWebsite where a visitor comes in interested in a product. The useraccesses the system with a click-through to the POC. Then, the systemautomates the process of going through the POC. Once they have conductedthe POC, the service is turned on, and they are now a paying customer.

To capture the user's interest, through the Website, the users areallowed to “try it,” for example. They enter their email address. Thesystem validates the email address with a first-level screening. Then,the system sends an email after they try it, and maybe a link to wherethey can see screens about how the system works.

In phase two, the system generates a demonstration room for them. Thisis based on some of the information they gave the system in the firststep, and in addition, the system now requires them to upload someinformation, a log file, for example, to provided the system with somehistorical information about how their Website has been used. The systemthen takes the log file, automatically generate a set of reports thatexplain to them what the expected increase in value the system canprovide. The system then goes through an automated process and creates a“before and after” picture of what their site looks like before thesystem and then after the system.

There is a certain amount of backend provisioning to do to make thatwork. Once they actually commit, in phase two, the system explains tothem that they need to upload their log files. The system can thenprovide them with a report that they can print out and use that to buildinternal momentum around the system. The system then allows them to usethe system in a POC fashion for some period of time, and then convertsthem to a real customer. The period of time could be 30 days, it couldbe 90 days, for example. Thus, this aspect of the invention takes aprospect through a methodical process online that requires very littlehuman intervention to allow them to experience the value of the system,without having to interact directly with a salesperson to feel anypressure, and without having to send a salesperson to their site.

Method of Alqorithm of Usefulness

This aspect of the invention concerns method that derives a score whichdescribes the usefulness of an electronic asset. In contrast towell-known relevancy algorithms, which help to find documents relevantto a query context, the computation of usefulness measures the actualusefulness of any electronic asset based on user behaviors with respectto the assets. Given a topic, there might be hundreds or thousands ofrelevant documents but only a few that are useful. Usefulness measureshow useful a document is for a given user, while relevancy measureskeywords that match with the content. Usefulness scores are computed forany electronic asset and for arbitrary user population sizes, rangingfrom millions to a single user.

Thus in contrast to traditional search technology, which is focused onrelevancy detection, the invention detects usefulness. With regard torelevancy, for example, if one is learning Java programming, there arehundreds of relevant Java books that can be used to learn Java. Are theyall useful? No. If one wants to really learn Java, one should ask a Javaguru what books to read, and they probably will recommend two or threebooks, instead of hundreds of Java books. Thus, relevant books compriseall these hundreds of books, while useful books are the two or threevery useful ones. This usefulness is based on the knowledge of experts,community, and peers.

Expert, peer, and community knowledge is automatically extracted andassembled by the present invention based on observed behaviors of theuser population. As user behaviors change over time, the system adaptsits representation of expert, community, and peer knowledge. Userbehaviors can be recorded in real time (through various means ofobservation described elsewhere in this application) or extracted fromexisting log files of user behavior. On an ongoing basis, the system cancontinue to improve performance, based on ongoing real-time observationsand getting continuing updates of the log files. The updates amount tothe differences in the log file, for example on a month-by-month basis.That is in addition to the information the system captures based onobservations. There are certain things that are in the Web logs that theobservations do not track and there are certain things that can beobserved in real time that existing web logs do not track. There is moreactivity than is needed in runtime or user time, but it is interestingto look at these after the fact and draw generalizations from the broadsets of data that are captured in these log files.

Method for the Self-learning and Adapting of Systems

This aspect of the invention concerns a method that enables the systemto identify changes in the behavior of its user population, with respectto both electronic assets and members of the user population themselves,and automatically adapts its operation to self-correct for changes.Self-correction enables systems to identify and adapt to changesproactively before they are obvious, while minimizing the need foradministrative intervention to keep systems maintained.

Thus, this aspect of the invention concerns an attribute of the system,i.e. inherited nature of system, because it observes peoples' behavior.As peoples' behaviors change, their preferences change, and their usefulcontent changes. The system automatically adapts to that change. Thus,the system is, by default, a self-learning system that can correctitself because, when people start to correct themselves, the systemfollows them.

The Inventive Technology is Content Type Independent or Content Agnostic

The system works against any content/information types, such asaudio/video files, data types, such as RDBMS data, and applicationnodes, such as a “buy” button. Thus, the preferred embodiment of theinvention comprises an independent and content agnostic system becausethe system does not look at the content itself. This is unliketraditional search technology, which parses content, picks up key wordsin the content, and uses those key words to select results. Theinvention, in contrast, is not concerned with what is in the content,but about the location of an asset and how people interact with thatasset. The invention does not care what that piece of content is. Itcould be a text file in a simple case, but it can also be a video file,which has no text to parse and no index to be built in the sense oftraditional technology.

The inventive technology seeds the system from Web server logs, searchengine logs, Web analytics server logs, and other log files so that itcan generate value from day one of the operation.

Supervised guidance can be accomplished through administrators byassigning experts and peers based on their roles, reputations, andexpertise etc., although it is not a necessary step Such information canalso be inferred and extracted from historical log files. Because thesystem is a learning system, it can derive more value over time aspeople use the system. This aspect of the invention concerns seedingtechnology that makes the system useful from day one. It may not be 100%useful, as it would be down the road, but it would give at least 50% to80% of the value. In this embodiment, the Web server log, which isactually a recorded history of what has happened in an enterprise, isused. It does not have the fine-grained information that is ultimatelyneeded, but it has coarse-grained information. The log file provideshistorical information. The preferred embodiment uses weeks to monthsworth of a log file depending on the site's traffic patterns. Thus, theinvention provides a way to take something a user already has, i.e. thelog file, and turn it into a resource that is used to seed the system.Then, over time, the system learns more because the invention is makingobservations by means of the extensions to the browser or the scriptsthat are running, as discussed herein. The system takes advantage of notonly basic logs, but also the analysis that is generated from those logsby higher order analytics which are available commercially from variouscompanies known to those skilled in the art.

The invention federates across multiple applications, Websites, andrepositories without crawling or indexing the applications.

The federation is accomplished through users' actual usage of thoseapplications. Federation is an attribute and a natural fallout of thecore technology herein. The traditional approach to searching is to havemultiple indexes, each of which is linked to a different repository ordifferent application. A search is performed against each repositorywith separate indexes for each repository that are not cross-searchable.

In the inventive system, a federated search is automatically providedbecause when people use an asset in a context of an application, they donot care where they use it. They can use one particular piece of contentin one sort of a silo, and the next minute can move into a differentsilo, e.g. start with a CRM system and then move into an ERP system. Inthis way, the user created a trail, i.e. a virtual link of the varioussystems. When this query is searched again, the inventive system canrecommend information from the multiple different data sources, suchthat that federation is automatic because the user is creating thefederation. That is, the user's pattern of usage of information from andacross various data sources creates the federation.

The invention herein does not require crawling of Web sites orapplications, or indexing of the applications or the contents thereof.Further, the invention respects any security that is already in place. Asignificant challenge in building federated search systems is thatfederated search systems must understand and work with the underlyingsecurity of these applications. It is difficult to do this because eachapplication generally has its own security model. Generally, securitymodels are not shared across different applications. The federation ofsearch while protecting security is a huge challenge. The invention isunique in the sense that it does this naturally, without any specificadapters, and it guarantees that it can preserve perfectly theunderlying security mechanism for that application. This is done in avery unique way. The system goes through the browser instead ofimplementing proprietary modules to preserve security.

Traditionally, to solve the federation problem, there would be some sortof search application that ties into each of the applications, and thatcomprises a specialized security model, conceptually. The problem is thesearch engine is actually building up an index of all of the content.When a search is performed, one cannot simply bring back a list ofsearch results and then prevent somebody else from clicking on the listif they do not have access to it. So, in effect, the search enginereplicates multiple security models in one index. The inventors haverecognized that there is no need to do this because the system has abrowser where a user queries through the system. The system thenaccesses its database of content and, in return, provides a list ofresults. The system does not filter out all the content at this timebut, instead, filters as the results are returned, The system providestechnology inside the browser that checks each of these repositories inreal time if this user in this session can access this content. If theanswer is no, the user is prevented from reviewing the content. It iskept off the list. The user does not even know it came up. The primarydriver for whether the browser has access is the person who is loggedinto the browser at the time, based on the person's privileges in thesystem, which determine whether the person can see the results. If theperson can not see some of the results, the system does not show theseresults. Thus, the system is, in real time, asking the application if aparticular user, e.g. the user currently logged in, can access thecontent right now. If this is true, let the user see the document. Thesystem, in real time, every time, asks the application if the user hassufficient privileges. Further, it does not matter what mechanism isused because the person can have different access rights depending onwhether they are group-based, identity-based, or profile-based.

Personalization, driven completely by usages of individuals, peer groupsand expert groups at non-predefined levels depending on contexts such asquery terms, navigation patterns.

This aspect of the invention accomplishes personalized search by knowingwho a user is thus, when the user exhibits certain behaviors when usingthe system, the user is self-identifying, e.g. through cookies, logins,etc. Even if the user is an anonymous user, the system places a cookiein the user's browser. Thus, when the user is using the system he leavesa personal trail, and the system then personalizes information based onwho the user is. In the system, no one predefines relations based onpersonalization because the system is based on the user's behavior. Theuser's affinity with other people creates a space, referred to as aclub. Thus, a user can form his own clubs implicitly by exhibitinginterest in one area. No one actually is monitoring the user. The clubsare established all through the user's behavior.

Controlled Deployment of the Invention for Risk Management andAcceptance Tests. The system reduces the product deployment risk bycontrolling the number of people who can see the product features in thelive running system. A special cookie is set and sent to a controlledtesting population. With that cookie, the users of the site can see theinvention's features while the general users have no visibility of thesefeatures. This is a desired way to deploy new technology in enterprises.

Augmented Search. This feature of the invention blends traditionalfull-text search with preference and activeness information from global,peer, and expert population to give users precise answers. This aspectof the invention states how the index is used. Using community insight,the invention can augment a search for a better result. In the augmentedsearch, a search request is made to the customer's Web server and aresult is obtained. Then, a request for search along with the Web serverresults, generating query, and user id is sent to search server. Aresponse comes back. Then, the system sends augmented results back insearch server format and the client renders the HTML.

Top N is a list of most useful information based on the context anddriven by the usage of the community. The context may be set by a userquery, explicit specification of topic, or presence on a particular webpage or group of pages, for example. The invention also creates aninsight called top ten, e.g. the top ten most important, related, usefulpieces of information, given a topic or given a context. The user cansee information based on context-driven usage of the information by thecommunity. Top ten is a popularity result. Give the user the ten mostpopular links that have to do with a query term (context), or maybe noterm. If there is no term, then the top ten most popular pages arereturned. For all of these views, one can apply a filter, e.g. only lookat the top ten that fall within the category of technology, or only lookat the top ten PDF files.

More-Like-This expands the content in similarity based on content useand community profile. If the user likes a piece of content, the systemobserves that and there are other pieces of content that can be shown tothe user based on the communities' interest. If the user likes a pieceof content, the system observes that and there are other pieces ofcontent that can be shown to the user based on the communities'interest.

More like this is a concept that applies when a user is reading thispage now, but wants to find another page that is very similar. More likethis is based on what the community says is more like this, meaningbased on the usage pattern. Thus, the community sees that this page issimilar to the page a user is reading

Predictive Navigation provides short-cuts for browsing navigation basedon where similar users start and end on an application.

In this embodiment, if people with the user's profile come to aparticular node in the application, then the user is highly likely to goto another, related place. This aspect of the invention predictsnavigation and shortcuts traditional navigation based on previousnavigation by peers and experts, including where they started and wherethey ended. Thus, the starting point and end point are critical topredict the user's navigation, to try to shortcut the middle part of aseries of navigational steps, and send the user straight to thedestination without wasting time in a lot of other places.

Predictive navigation is also referred to as “Next Step,” and depends onwhich calculations or results one wants to display. Predictivenavigation uses the navigation trail. There is a notion of a navigationtrail; the system tracks the last N pages a user has been to. The systemkeeps a history of the query that was used to start this trail, ifapplicable. Thus, the user searches and a result comes up. The user mayclick on the result. The user may click on the result. The user mayclick again to go somewhere else. The user keeps clicking. The systemaccumulates this history of pages. It also notes that this entirehistory was due to a query. The system tries to, based on the user'shistory and other observations in the past, figure out where the user isgoing. The recommendations that come back are pages ahead of the userthat other people have found useful, based on the history, i.e. thetrail that the user has accumulated in this session. The system thustries to match where the user has been and figure out where he is going.

This aspect of the invention states how the index is used. Usingcommunity insight, the invention can augment a search for a betterresult.

Zip through concerns the idea of having content preloaded on the system.As the user is going down the results, the system shows the user apreview of what that link is. Thus, instead of having to go to the pageand changing the page that the user is viewing, the user just zipsthrough. If he sees the content he wants that is the one he clicks into.

Dynamical and adaptive identification of peers and experts in the usercommunity.

The peers and experts are not hard drawn circles but a web with hubconcentration and connections among various hubs. Information andknowledge is aggregated together and a fusion effect of wisdom iscreated naturally and automatically.

The invention essentially uses the same information to identify the usercommunity. Who are the peers? Who are the experts? Not only does theinvention identify what content are user would like to see, but also itcan identify the actual people who are the user's peers. The grouping ofpeers is naturally formed. There are no hard boundaries on them. Thereare affinities in these people.

Implicit voting that is unbiased and critical.

This aspect of the invention provides far more accurate prediction ofcontent usefulness than traditional content voting or survey. Documentrating driven by usage reflects the real value of content. For a systemsuch as the herein disclosed system to work reliably with highconfidence, implicit observation is very important. If you ask people tovote on content, you tend to get biased results. You also get a samplethat is highly skewed because most people do not have time to vote, tosurvey, to do anything that is explicit. People who do vote tend to haveextreme opinions. They have a lot of time on their hands. They areoutspoken and opinionated. Thus, they tend to misrepresent the entirepopulation. The sample does not match the population. They also tend tovote negative more than positive. Thus, the invention preferably doesnot use explicit voting. It takes account of implicit actions. The userrequesting a print is implicit because he's doing something else at thetime he's making the request. The user is not giving feedback and notbeing asked for feedback. The invention exploits passive observation,not active feedback. Although, some embodiments could include activefeedback, such as negative feedback.

Method for the Computation of Wisdom (Relative Value of Assets to a UserCommunity Over Time)

This embodiment concerns method that observes information aboutelectronic assets, the behavior of the user population with respect toelectronic assets, and the changes in assets and behavior over time todiscern the relative value of the assets to the user population overtime. The method identifies a spectrum of values ranging fromshort-lived Information, to mid-range knowledge, to long-lived wisdom.Ultimately, the system provides an on-going, content agnostic, andadaptive institutional memory for the enterprises.

Computational wisdom means that the wisdom is a form of communitybehavior that, with regard to a set of assets, does not change overtime. There are four items stated above in terms of how frequentlypeople change opinions. Content, for example, is the least reliablething to trust because content can change. Information is at a secondlevel of trust. If information stays stable in view of people'sopinions, that set of information becomes knowledge. If knowledge can gothrough time and continue to be used and supported, then that becomeswisdom. So, wisdom cannot change year to year. Knowledge may change frommonth to month, and information may change from day-to-day. Content byitself does not mean anything. Thus, if content does not become lessuseful over time, but it becomes constant as time goes by and usefulnessremains constant, then it passes from content to the stage ofinformation, then to the stage of knowledge. And over a certain periodof time if the usefulness remains high or continues to increase, itbecomes wisdom. Whereas, if there is fluctuation in the usefulness overtime, then the change shows that maybe it is not really wisdom, but justcurrent information that is interesting.

The invention provides content gap analysis through the use of content,data and application instead of relying on content owners' speculationon what's missing, hot, good, and bad.

One does not know what content is missing, or what people are lookingfor. Also, it is not known what kind of content that can be producedcomprises that which people need to consume. Because it is known fromthe system what the trends are, what people are asking for, the questionof whether they are being satisfied with some content can be answered.That is, it is known where the gaps are. A lot of people are requestingthis thing and are not finding anything useful. There is a gap.

The gap analysis report provides the ability to detect gaps in thecontent. The assumption is the content is there, they just can not findit. Frequently there are gaps. For such cases, the system ascertainswhat people are actually looking for and what is missing. Someone in atraditional search or navigation situation might search for something,and then either they fail, or if they do not resign to failure, they maysearch again, or they might potentially try to navigate. Through eitherof these mechanisms they might have success or failure. The systemaddresses this problem when someone starts to exhibit search ornavigation behavior. It is known precisely what they are looking for,and the content that over time starts to get surfaced in search andnavigation is the content that the community, itself, has deemed useful,without regard to a developer or merchandiser, or what merchandisersthought about the content. Thus, somebody is looking for something, theythink it is going to be useful, but they are not finding it. This aspectof the invention allows one to quantify how dire the need is for thisinformation.

Applying the information gaps at a division, company or industry level.

The system provides the information flow over time, and helps company tomanage information logistics. This aspect of the invention uses anapplied information gap as a division flow over time to identify howinformation flows. The system allows one to understand what people arerequesting and what content is available that can meet those needs. Intime, it is possible to see the flow of information going in and outfrom division to division, from location to location. It is possible tosee which location presents what information, or what group, or for whatnumber of people.

The ability to identify experts and peers to enable companies to locatetheir expertise at a global scale.

A dashboard for company or industry efficiency against informationconsumption can be measured, and white-collar workers productivity canbe derived for the first time. This aspect of the invention is relatedto the ability to identify experts and peer-enabled companies with theexpertise at a global scale, which allows the system to provide adashboard for finding out who knows what and what can be done where.

Method for Building Automatic Marketing List Through SponsoredAdvertising

This method identifies firms that are purchasing keywords and banner adspace on public websites for advertising purposes. The invention looksat both common and uncommon keywords, as well the context of givenbanner ads, and automatically generates a list of firms who areprospects for improved lead generation through their websites. Theinvention uses information found in the online ad itself and combines itwith other public information sources to create a refined list of firms.The system then back-traces to the buyers of the ads, and automaticallyincludes that information in the candidate prospects list.

Method for Improving Sponsored Advertising Conversion Rates

This aspect of the invention helps firms who wish to retain customers orincrease lead generation. This is accomplished by increasing theconversion rate of sponsored ads e.g. Google and Yahoo ads.

Based on context and where a user came from, e.g. from Google with agiven search term, the system can guide this user to the most usefulinformation on the given website or collection of related websites.Without this capability, users who arrive at website no longer have thebenefit of these public search engines directing them to the mostrelevant information.

The invention routes these users to the most useful information byobserving where the community found the most useful information, giventhe initial query context from the public search engine. There are twosteps in this process: 1) first, the system captures the collectivewisdom on where useful information lies given the query context; 2)secondly, as users come in from Google or Yahoo, the system leveragesthis collective wisdom to guide people to exactly the right content.

Engineering Features

Usage-driven link analysis

If a link is clicked by a user in a particular site, the text withinthat link is then captured to augment future searches and navigationwithin that site. The text is noted only if the navigation trail leadsto a successful document discovery, i.e. the user finds the documentuseful implicitly.

This aspect of the invention, usage-driven link analysis, concernsanchor text. This is very different than Google because when Google ispage-ranking they parse every single page, and how many links, and otherthings. The invention parses links that are used by people. A link is adead link unless it is used. So, if someone clicked on a link, then thislink is useful by this person.

Furthermore, used links are cross-examined by many uses of peers andexperts, in addition to that of the individual users. The peer andexpert implicit validation and endorsement reduces noise from individualbehaviors and strengthens the signal-noise-to-noise ratio.

The successful use of a link is determined by capturing and analyzingindividual user behaviors, peer group behaviors and expert groupbehaviors, with respect to that link.

Usage of a link and the importance of the text is determined by ablended vector of user, peer, expert, query context, and time. Thisaspect of the invention, successful use of link, determines how anindividual user behaves. In addition to looking at a link itself, whereif a user clicks on it, it is useful, the system also does additionalanalysis on how many other peers similar to the user click on theselinks. For example how many other experts are different than the user,but the user depends on them to do his job, and who also clinked on thelink. Thus, there is a two-level value: individual use of content, andthat of the peer-group and expert group. These dimensions give the totalvalue of the data.

Implicit Building of Context Terms

It has been historically challenging to create metadata for content. Theinventive system deploys a unique way of letting the communityimplicitly create context terms that describe what the content is about.The context terms are learned through watching users conducting queriesand finding useful results via the queries. It also includes navigationtrails with the link text associated to the each use. The system buildsits vocabulary by watching how visitors uses various terms to describecontent and how a site uses links to describe content. Again, more usedqueries and links are more important and associated with content, whilea link text that yields no use of a content in the downstream trails hasno association to the content.

Capturing Such Information is Done Via One of Three Methods

JavaScript Tags. In this method, the page is instrumented with a pieceof JavaScript. This JavaScript detects link usage and text and sendsthis information onward to a server.

Browser Add-on. In this method, the browser is instrumented with a pieceof software. This software detects link usage and text and sends thisinformation to a server.

Log Analyzer. In this method, the access logs for a web site areanalyzed via a special program—the Log Analyzer—which detects usage oflinks and sends this information to a server.

All the information captured above is referred to as observations. Theanalysis of observations captured above takes place in the server.

Client

The system client comprises three general areas: the UI, the observer,and the proxy.

The client comprises a Web browser that entails the client UI (see FIGS.5-7). The client includes a JavaScript observer to make observations onusage of the Web page at the client. One embodiment of the inventioncomprises a sidebar UI that shows the recommendations from the systemengine. This aspect of the invention is embodied as JavaScript tags thatgenerate the JavaScript necessary to display the UI. In this embodiment,enterprise Web content is displayed on the page and along the side thereare system generated recommendations. A variant of the UI provides apopup, where the user clicks on something on a page, e.g. an icon, thatcalls into system code to display a popup in context.

The UI also comprises an API for fetching results from the systemserver. This is in lieu of gaining results directly from the enterpriseinstalled search server. On the typical enterprise site the user typesin the search term, clicks on search, and the search hits their Webservers. The Web servers then go back to their search server. The searchserver returns the result in some format, usually in XML, and then theyhave a presentation code on the front end to parse through the XML andpresent the search results. The invention operates in a similar fashion,except when their Web server goes back to the search server, instead ofgoing back directly to the search server, it goes back to a server sideextension. The extension then fetches the initial results from theirsearch server, feeds that back to the system, and the system eitherreorders the results or, in any case, enhances the results, possiblyadding more entries. This is provided back to the extension, and theextension reports back to their Web server. Their Web server continueson as it did before, parsing through the XML, reformatting, and sendingit back to the client.

The JavaScript observer is a piece of JavaScript code packaged as a tagthat is given to the user, and the user instruments their page usingthis tag. The JavaScript tag resides on the client page and makesobservations. For example, a scroll or a dwell observation. If theend-user, for example, is reading a page, he would conform to what isdefined as a “dwell.” Once a dwell occurs, i.e. once the JavaScriptobserver has observed a dwell, it then sends back that information tothe server. The server accumulates these observations.

Each of these observations correspond to a particular calculation on theback end, e.g. at the affinity engine. The augmented search concerns thenotion of reusing the UI that the user has, instead of standing inbetween the presentation layer and the search server and augmenting thesearch from there. Predictive navigation, top ten, more like this,context expansion, and zip through are all types of tags that the usercan put into a page, and they all use a different algorithm in creatingsuggestions.

The recommendations that come back are pages ahead of the user thatother people have found useful, based on the history, i.e. the trailthat the user has accumulated in this session. The system is thus tryingto match up where the user has been and trying to figure out where he isgoing.

Observations

There are eight directional observations coming from the browserextension, which is a script that is watching the user's sessions andcollecting information.

The proxy demonstrates the system UI on a user's pages where the systemdoes not have access to the source code of those pages. These users areprospective customers who do not want to give out access to their pages.The system wants real-time pages, but with our tags injected into thepage to show the user what the page would look like with the systemenabled. To do this, the system uses a proxy that goes and fetches thepage from the URL and then, based on a configuration of rules, altersthe HTML of that page, and then sends that page back to the browser. Theproxy sits between the browser and the target Web server, i.e. thepotential customer's Web server. The proxy itself has its own URL, andit just passes in the target URL as part of that URL. A URL for purposesof this embodiment of the invention consists of two URLs. The URL usedactually points to the proxy, but embedded in the URL itself is thetarget, i.e. the customer URL that you want the proxy to go to. The URLgoes to the system first and reconstructs the URL, bringing you to thecustomer page. Then, the proxy makes an HTTP connection on your behalfto fetch the page for the customer site. It looks at the page andapplies a set of rules to it, and then sends the page back to the user.

With regard to the client browser, the page is first instrumented withtags. The presently preferred format of the tag is in JavaScript (JS).The customer incorporates a JS file on their Web server. Then they referto it in HTML with a script tag. Once the JS file is loaded, the filesets up an object at the system. Then, a place is set up in the pagewhere the UI is displayed. On the system side, the system sends backHTML to the UI. The administrator on the customer side specifies a stylesheet on the tag. Even though it is the same HTML, because the stylesheet is different, the user gets a different color scheme and font, forexample.

A plug-in may be provided that serves a similar purpose as the proxy, inthat it also modifies the HTML and comes back with a search result.However, unlike with the proxy, the user configures the plug-in. When asearch of the URL is performed on the user's internal site, the plug-intakes the search request, performs a search for the results, sends themback to the system, which then augments the results and sends them backto the plug-in. The plug-in does a swap of the HTML that is displayed.Thus, instead of displaying the HTML of that URL, the plug-in displays amodified page from the system. The plug-in also makes observations.Because it has more access to the browser functions than JavaScriptdoes, it has a better ability to capture a wider range of observations.

The variant UIs work the same way. For example, Top Ten and predictivenavigation work the same way as discussed above for the UI. The onlydifference is in the request. For example, when the user asks the systemfor augmentation, the system is asked for a specific calculation.

The JavaScript observer sits and waits and observes the user on thepage. An observation is made of a user action and the observation issent to the system, including all the information about the observation,e.g. what page it is, if there is user information.

Dwell is observed when the user has spent N number of seconds or minuteson a particular page.

Range can be selected as a matter of choice, but is typically around 30seconds to five minutes, depending on the complexity of the documentthat being inspected. An excessive amount of time means the user walkedaway from the computer The system preferably does not capture an upperthreshold because the user may be reading the document.

There is a virtual bookmark/virtual print feature where the user readsthe document, and finds it useful, but can not remember everything thedocument has said, so he leaves the window up somewhere behind his otherwindows. The user then goes and does other tasks, and when he needs torefer to the document, he pops it back up; the user can tab document.Thus, even though the user did not bookmark the document, meaning thatthe information is probably useful for the next few hours or day or so,and the user does not need to bookmark it, it is useful for the user tokeep the document up the window. In such case, the user has notexplicitly bookmarked a document, but he has left it open for a reason.If one looks at a typical computer at any given time, the things thatare open on the computer are not open because the user was done withthem and just did not close them. They tend to be open because they area virtual bookmark. Thus, things that are left open for a very longtime, e.g. two minutes, five minutes, ten minutes, are considered to bevirtual bookmarks.

A scroll concerns a scrolling of the screen.

The anchor text is the hyperlinked text that got the user to the presentpage. It could be as simple as the user clicking on a news items thatthen brings up some recent news about the subject.

Think is a usage pattern, i.e. a combination of a dwell and a scroll, orsome action, mouse movements, etc., that indicates that the user isthinking about the page. Thus, think is a period of inactivity followedby some action.

Mail is when a user mails or forwards the content to another user, orvirtually emails it in a similar fashion of virtual bookmark with theintent to mail.

Affinity Engine and Wisdom of the Community

FIG. 12 is a flow diagram showing document recommendations according tothe invention. At the beginning of a search 110 various term vectors T1,T2 exist, as well as a peer/expert population. A term vector isavailable and is compared with the term vectors of every other documentsuch that the top N matches are selected. The most popular terms for theN most documents are found and these are added into the term vector aswell. At this point, activeness information may be obtained for everydocument and the new term vector can be compared to the term vector ofevery other document. The two can then be combined 118 and the top N canthen be selected 119. The concepts of term vectors and documentsearching are discussed in greater detail below.

The invention uses a similar strategy to the way in which a search isperformed inside of a document. The known approach represents everythingin a vector space model. This approach takes a document and makes avector, which comprises all of the document terms in term space. This isdone for every document. The search is represented as another vector interm space.

In this regard, the invention uses a vector space model, but the way itbuilds the vector to represent a document has nothing to do with what isin the document. It has to do with what other people have searched onand have hit this document. For example, an individual performs a searchon the term “chip design” and perhaps other words. He might have endedup finding a document. It might have been low down on the list ofreturned results, but he might have ended up finding it. As soon as hedoes, and he finds it, the invention then associates whatever hesearched on with that document, and that contributes to the vector.There are other ways to fill out the term vector, e.g., through a usersnavigation behaviors (described later), or through explicit input by theuser. Thus, the invention builds representations of the document basedon how other people are using it.

Instead of having single-term vectors for a document, which is whathappens in the search space, the invention gives every individual usertheir own term vector for a document. Accordingly, every user gets tosay what their opinion is on what a particular document is about. Somepeople may have no opinion on certain documents. However, if somebodyhas ever performed a search and used the document, for example, theiropinion gets registered in their vector for that document. Knowing this,the invention allows various functions to be performed, such as “I wantto match all the documents, but I don't want to look at everybody'sopinion;” or “I want to look at just my peers' opinions or the experts'opinions.” In this case, the invention takes several of the term vectorsfrom different people, sums them together, gets a view of what thatpopulation thinks that document is about, and uses that result toconnect people to the right documents.

Thus, this aspect of the invention provides a search enhancement thatrelies upon the novel technique of a usage-based topic detection. Theterm vector provides a vector space model to represent usage-determinedtopics.

Activeness. In addition to the term vector, the invention also comprisesa vector that looks at what documents each user uses. Every user has oneof these, called the activeness vector. Every time they use a particulardocument, the invention notes that they have used it. Every bucket inthe activeness vector is a particular document or asset, and the bucketkeeps an accumulation of usage. Some buckets may have zero. Some mighthave a huge number. Different actions on an asset by a user, e.g.,reading or printing, contribute differently to the activeness vector. Anactiveness vector for a population can be generated based on theactiveness vectors of each of the users in that population. For example,for a particular population, e.g. seven users, the usage vectors aresummed to determine how much a particular document is used in thatpopulation. The invention combines these two pieces of information, i.e.the term vector and the activeness vector, to help recommend documents.For example, there might be a document that matches perfectly in termsof topic but is not used very much. Even though both vectors concernusage-based information, one vector concerns an amount of usage and theother vector concerns the context of usage. The invention brings thesetwo numbers together to suggest a document. The invention can alsoincorporate the results from an existing search engine (which amounts totopic match based on the contents of an asset) with the term andactiveness vectors if we want.

Each individual has his own library (collection of used assets) and foreach document in an individual's library, the system has the user's termvector, which represents what they think the document is about. Thesystem also has their activeness vector which indicates how much theyhave used that document in any context. It is now possible to bringtogether any given group of users and ask for their collective opinionon a document. There are also the global usage vectors, which are thesum of everybody's vectors. There are also special usage vectors, forthe group of anonymous users. Everybody who is unknown all contribute tothe same usage vectors. When combining vectors to create a collectiveview, there can be a different weight for different people's vectors,but the sum of everybody is the global vector. The invention alsocomprises users' peers and experts.

Peers. The way that the invention determines peers is similar to the waythat peers are determined in collaborative filter applications, such asAmazon.com, but in this case the determination is based on documentusage. The invention looks at two user's document usage (activeness)vectors, e.g. one person has used this document, this document, and thatdocument; the other person has used the same three documents, thereforethey're similar. That is, there is a document usage similarity that, inthis case, is established when the invention compares the two user'sactiveness vectors. In the preferred embodiment, two users that overlapon a significant set of used assets are considered to be peersregardless of other assets used only by one user or the other. That is,the fact that one user uses a subset of what the other uses means theyshare a common interest or role. In this regard, the invention can alsolook at the actual terms that people use: do they search on similarterms? Similar documents? Similar search terms, or a blend? Thus, theinvention considers term usage. Another consideration is the user's areaof expertise. Consider, for the moment, that two people have expertisevectors, and that their expertise vector is a term vector as well. It isa term vector that, instead of representing a document, represents aperson and what they know. It could be from a profile or it could beautomatically detected based on what they use.

Expertise. What a given user knows the most about is his/her expertise.The system can automatically determine a user's expertise based on whatassets they use and what topics they tend to spend time on. Expertise isvalidated. The invention looks at a person's collection. We ask theglobal population what they think those documents are about, not whatthe user said they were about when he searched for them, which is theuser's term usage, but what the global population says these documentsare about. When looking at the combination of global term vectorassociated with the assets in a given user's collection, a pictureemerges as to what that user's expertise (which can also be representedby a term vector). A user can not self claim what he expertise is. Thepopulation of other users ultimately determines the expertise vector. Toidentify an expert, the system looks at the expertise vector. Forexample, if a user is searching on chip design, the system looks atevery user's expertise vector, and finds out who is most expert on chipdesign. The system selects, e.g. the top 30 experts. The system thenfinds those 30 experts' term vectors and activeness vectors for everydocument, sums them together, and then performs a comparison.

An alternative and complimentary approach to the determination of userexpertise first identifies those documents in the collection that havehigh impact factor for the given topic of interest. Asset Impact Factoris a measure of how useful an asset is to a particular population, suchas the global population, for the given topic. Once the impact factor iscomputed for assets in the entire collection, every user's library canbe assessed in terms of impact factor of included assets. Using thismethod, users with relatively many high impact assets in theircollection of used assets are considered to be experts on the giventopic. Such users may also be assigned an Expert Impact Factor,reflecting the impact of that user's asset collection on a givenpopulation for a given topic.

If the user is on a particular document, using a term vector for thedocument based on either the user's global population, peer population,or expert population, the user can ask what documents are similar tothis e.g. ask for “More like this.” The system can compare thisdocument's vector to every other document's term vector. In this case,the invention is looking at the term vector, which is determined by agroup as to the relevance of terms to a particular space. Therefore,there is a second measure on top of the term vector, which is themeasure of relevance. It is therefore possible to say that this documentis relevant in this space, not just that it has these words in common.

When a search is performed using the term vector and the activenessvector, the user gets the most useful documents returned to him. Thesystem can also say that now that these documents were found, thevectors are known, and it is possible to go off and find additionaldocuments that might be of interest and suggest those. This is a way toexpand the list of search results. The context in which this happensmost often is when the user is navigating around, and has found acertain page. The user opens the documents that are closest to this one.The user clicks on the document that he likes, and then says that thisis close, so show me more like this. What gets returned may, in fact,have been in that original search list, but there might be some newthings as well.

The system performs navigation tracking based on usage in auser-specific way. Thus, the invention can also track where people gofrom one document to another. A user may end up going to a particulardocument looking for information, but may then have to click throughseveral documents before finding the useful documents. If one or moreusers follow this same pattern, the system can recommend that a userlanding on the initial document go immediately to the useful document.That is, the invention makes an association between documents that arefound to be useful, even where other documents may have been encounteredby users along a navigation path.

Thus, the invention recommends going straight to a document usefuldocument, without having a user navigate through documents that maynormally intervene. This is based on usage: for each user there is amatrix representing connections between visited documents and the mostuseful documents arrived at from that location. As in other aspects ofthe system, we can combine user opinions to get a collective opinion,e.g., of the user's peers, experts, or the global population, in thiscase regarding where are the most useful places (assets) to go to fromthe current location. For every user, for example, we can take each ofthe peers' matrices and add them all together, and then come up with arecommendation. In this way, the invention keeps track of the navigationpatterns of the user's peers. In addition to providing recommendations,identified navigation patterns can also be used to providevisualizations of user activity within an asset collection on a globalor population-specific basis. Such visualizations or usage maps can, forexample, be very informative to site designers in understanding theeffectiveness of their site and understanding user interests.

One concern with a system such as that disclosed herein is that as thesystem starts recommending documents, it tends to reinforce theirusefulness. The invention counters this using a validation technique.For example, if there is a fad that the system is reinforcing, e.g.people go to a particular document all at once, but they are not goingto return, e.g. a fad movie. Everybody goes because they hear that it isgood, but in fact, it is terrible and they hate it. A lot of people goto the document, but no people come back to it. The invention adjuststhe usefulness of the document by looking at the percentage of peoplethat come back, and determines if there are enough people coming back tovalidate it as a legitimately useful document. As for making adetermination, in one embodiment if something is new and it does get alot of attention in the beginning, the system encourages that attention,in a sense, because it may be something that is new and important. But,if over time, people do not start coming back, it is going to decay awayquickly. Accordingly, the invention considers both the notion of newnessand the notion of validation.

Besides connecting a user from document to document, the invention alsouses navigation to find information that identifies what a document isabout. When somebody clicks on a link and goes straight to a documentand uses it, that tells the system that the user clicked on this linkthinking he were getting something and then he used it. Whatever thelink text is, it is a decent reflection about what this document is. Thesystem then uses that link text and now that link text also contributesto the term vector. It is as if the link is a substitute for a queryand, in fact, if the user has clicked on, e.g. ten links going all theway through this document, the system uses the link text, with someweighting because not every link has the same weight. It depends on howclose the link is to the document. If the user clicked on one word in adocument and clicked on another link, then another link in the nextdocument, the most recent link gets the most weight, and earlier linksget less weight. Thus, in this embodiment weighting is based on a formof proximity.

Another aspect of navigation addressed by the invention concerns where auser starts with a particular document, e.g. document 1, and the systemmakes various recommendations based on this starting point. In the eventthe user was first on a different document, e.g. document 13, than ondocument 1, the system may recommend a different set of documents. Inthis case, the invention employs an additive model which look sat thedocuments that the system recommends from, e.g. document 13 and looks atthe documents it recommends from document 1, and the system weights themtogether. In this way the system may use the user's navigation trail(encompassing one or more navigation points) to suggest the bestlocation to go next. The person skilled in the art will appreciate thatthere are so many options available for processing the systems vectorinformation.

One aspect of the invention concerns determining what is a successfullyused document. One approach looks at how much time has somebody spent onthe document. There are two concepts related to this: One is the idea ofdocument processing time. What we would like to be able to infer is howmuch time somebody is actually spending looking at and reading thisdocument. This is in contrast to not reading the document at all, whereit just might be on the screen, but the user is not looking at it; orthe user is seeking for something, searching, but not finding it.Document processing time is the simplest measure of successful usebecause the system only need look at how much time somebody's on adocument. Another consideration is that seeking time is not the samething as processing time. In this case, the user is scrolling around andnot finding what they want, i.e. they are not processing it. The systemcan take the time spent on the document, subtracting out the time thatthe user is scrolling on the document. The system can also use scrollingas an indication that the user is actually on the document. The systemcan apply a combination of scrolling and pausing and use that to get asense of how long the user is actually processing the document. So, ifsomebody scrolls, then they stop, and they sit there for 30 seconds,then they start scrolling again, the system can make a guess that theuser was reading that document for 30 seconds. Any mistake in the guessis an aggregate because the system rarely looks at just one person'sopinion, but is summing this information over a group of users.

The term vector is a big vector of every possible term. In the presentlypreferred embodiment, the system increments a term's vector entry eachtime the term is associated with a document through user behavior. Termsthat are not associated with a document get a zero. On top of this,terms that are associated through usage with many documents get less ofa weight in the term vector because they are very common. Thus, thesystem looks at all the terms that are known and looks at how manydocuments those terms are associated with. Terms that are associatedwith many documents have a count for the number of documents that theterm is associated with. The system applies a formula to lower theterm's rating based on its association with many documents. If there isa word that is in every single document in a collection, then it'srating is equivalently a zero. Certain words, such as “the” and “and”are standard stop words and are removed from a search before they evenget to the system analytics.

Thus, the system commences with the creation of the initial term vectorbased on a data structure. For every user in the system, there is amatrix of documents and associated terms referred to as a term docmatrix. This is the collection of a user's term vectors for eachdocument known to the system. In other words, every user has a term docmatrix that represents for each document, what that user thinks thedocument is about. For example, one person thinks a document is aboutoil and about refineries, but not about something else. The systemservices a particular population that is already selected, which cancomprise for example peers or experts, e.g. the person's top 30 peers.To perform a comparison, the system knows of these 30 users, and each ofthese users has a weight based on how much of a peer they are to acurrent user. The peer with the greatest weight is the top peer, thepeer with the next greatest weight is the next best peer, on so on. Theinvention looks at all of the term doc matrices of these peers and addsthem together given the weightings based on their peer scores, andproduces therefrom a single term doc matrix which is this population'sopinion on every document in the system. The system then takes thismatrix and calculates a cosine between a term vector representing aquery or current context and each of the rows in the matrix, whichrepresent term vectors for each document. The result of the cosinecalculation represents how closely a document matches the contextaccording to the user's peers.

Once the system has determined all of the term vectors in the populationand assigned numbers to each document, the system then selects the topdocuments. One way to do this is to select the top ten document and thensum them together to get a single vector which says in the aggregatethat these documents are bout a certain topic. Then, the system takesout those search terms that are already used and looks at where theother peaks are. Those are additional terms that the system either wantsto suggest or automatically enter to the core. Now there a new termvector, and the system goes through the same process. The system thencan match this new term vector with every other document, get a new setof scores, and select the top ten of those documents. The system has amatrix that has everybody's opinion on what these documents are about.The system can now compare this new term factor and get a new set ofscores for all of them. The system also goes and we get a single vectorfor each document which indicates how related this document is to thetopic with another score, which is how useful this document is.

Once the system has determined all of the term vectors in the populationand assigned numbers to each document, the system then selects the topdocuments. One way to do this is to select the top ten document and thensum them together to get a single vector which says in the aggregatethat these documents are bout a certain topic. Then, the system takesout those search terms that are already used and looks at where theother peaks are. Those are additional terms that the system either wantsto suggest or automatically enter to the core. Now there a new termvector, and the system goes through the same process. The system thencan match this new term vector with every other document, get a new setof scores, and select the top ten of those documents. The system has amatrix that has everybody's opinion on what these documents are about.The system can now compare this new term factor and get a new set ofscores for all of them. The system also goes and we get a single vectorfor each document which indicates how related this document is to thetopic with another score, which is how useful this document is.

Consider an extremely popular document, a very unpopular document, asomewhat popular document, and a document that has never been used.Every user has a document vector or their activeness vector, whichidentifies what documents that have been used. Each user also has anassociated weight based on the peer population. Given these two numbersfor every document, i.e. how well it matches and how popular it is, thesystem combines the two to produce a score. One way to combine them isto calculate a weighted sum. Another way is to take the numbers as is,but anything that is below a threshold is removed. In the firstapproach, when the system adds them together the orderings of thedocuments is changed; in the second approach, the ordering that getspreserved as the ordering of relevance, but the system is removingthings that do not meet a certain threshold of accuracy. In anotherapproach, instead of a straight combination, apply a transform whichtends to favor more balanced numbers than it favors extremes. A straightlinear combination favors extremes but, for example, a square rootcombination can produce a more balanced result. Once the system hascombined the vector, there is a ranking, and the system can recommend,e.g. the top 10 documents. Once the documents are returned to the user,the next part is watching what the user does with the documents. Forexample, the system looks at such actions is the user navigating in acertain way, staying a certain amount of time on a document, where arethe links going. The system collects that information to learn about theusefulness of a document. The user starts by doing a search on a topic,e.g. oil. The system responds by recommending certain documents. Theuser clicked on one and printed it. In the user's term doc matrix, thesystem adds a number for that document connected with those words. Ifsomebody else does a search and the term doc is involved, the matrixindicates that the document is relevant for a certain purpose accordingto a certain person. Thus, if the person is a close peer and thedocument is a relevant document, the term match is good, and thedocument is recommended. If the user is not a close peer or the documentdoes not have a good term match, or the document is not very active,then it is not recommended.

As discussed above, there are two vectors, which express usage-basedrelevance and activeness. There is also a third vector that could beproduced when the two vectors are combined with search engine results.Thus, these three things are combined into this vector. There are manyways they can be combined to determine how much to weight the searchresults from the search engine versus how much to weight the otherresults. One approach is to obtain a list of IR results and, from thatlist, remove everything whose accurateness is below a certain threshold.This produces an augmented search, where the search is augmented byremoving the less accurate results.

Another approach involves separate vectors related to peers, experts,and the global population, which are then combined with differentweightings, e.g. the experts get the greatest weighting, the weightingcould be profile-based, or it could be based on user responses to aseries of questions.

As discussed above, every user has a term doc matrix that captures whatthe user thinks every document is about, and they have an activenessvector that expresses how much the user has used these documents. Thisactiveness vector is not only used through search. It could be usedthrough navigation and is built up based on search terms or link terms.To determine peers and experts proceeds as follows:

For a given user, build a picture of what that user's expertise is.Validate that expertise by the global population or by the appropriatepeer group. In the first case, the global population has a term docmatrix which represents what the global population's opinion is aboutevery document. This is essentially a sum of every single user's equallyweighted opinions on the document. This is a global opinion. For eachuser, look at what documents that user has used, and that they have usedthem a certain amount. This step involves determining the expertise ofthis user. For example, this user has used document one at a weight of,e.g. four, so when the system goes into document one it determines whatthe global population think this document is about. Take that, multiplyit by four, and add it to the user's expertise vector.

If the global population thinks that is an important document and if theuser has used it a lot, the user has more expertise, The system doesthat for every document in this user's collection. The things the userhas used the most get the most weight in terms of their expertise. Eachtime the system adds what the global population thinks about thedocuments that the user has used. Thus, expertise is a measure of whatexpertise does a user's collection represents. The system does not knowwhat a user's actual expertise is. It could be somebody who has done anexcellent job of collecting all the right documents on this topic, butif he's done that, in a sense, he serves the purpose as an expert. Thatis, if the user has all the good documents on that topic, therefore thatcollection is an expert collection. The amount of weighting to give thepopularity of documents is an issue. An amount of weight is given to howused a document was by this user and an amount of weight is given to howpopular the document was in the population. The system combines thesenumbers and recalculates expertise, every night for example. Thus, thesystem recalculates everybody's area of expertise because it mightchange on some basis, e.g. a daily or monthly. The system goes throughand calculates everybody's area of expertise, and then if it is desiredto figure out who the experts are, given a particular query, the systemtakes that query vector and compares this query vector to every user'sexpertise vector. Then, the system can produce the top N of experts, andthat is the expert population. Another case occurs where the system doesnot have a query and has a document but the user wants to know who theexperts are. In this case, the system can use the document itself todetermine who the experts are. Thus, the document itself has a vector,and the system can compare the vector of this document to the expertisevector of everybody and, given the topic of this document, determine whothe experts are on this topic represented by this document.

Peers. Every user has a term doc matrix and an activeness vector. Thereare three things that the system can look at and combine to determinepeer-hood. One is to compare what the peer value is for, e.g. two users.The system makes this determination for everybody, but for now focus ontwo users. Look at one user's activeness vector and another user'sactiveness vector, and look at how similar they are. Two people that usesimilar documents a similar amount are similar, looking in the sameplaces and at relatively proportional amounts. In this case, there is asimilarity metric between one user's activeness vector and anotheruser's activeness vector. Another way of determining peers is to look atwhat topics they are interested in. To do that, sum their term docmatrix, which gives a sense of what topics they have searched on andused in the past. The sum represents what this person is interested in,and is referred to as an interest vector. The system compares interestvectors. Thirdly, the system can compare the computed expertise vectorsof each user to determine peers. Alternatively, the system could employcombinations of these approaches. Because the user has a particularsubject in mind and the user has a certain number, the peers areweighted according to how closely they match that particular person.Some peers may have a closer number to the user's number. Some are goingto have one that is smaller or larger, depending on what notation thesystem is using. In the end, there is a number that' indicates how muchlike this person the user is. The user might want a peer group of 30,then number 30 in the group might have a smaller weight, and number 1 inthe group might have a greater weight, and everybody's in between have aweight between. The system could also have a threshold that does notcreate any peers less than the threshold.

FIG. 12 is a flow diagram showing an augmented search according to theinvention. In an augmented search, a search request is made by a clientof customer libraries. The search is sent to the search server and theextension makes a request for augmented information, for example fromGoogle. The augmented results are returned to the server and the resultsreceived are added to the server information which are then sent back tothe search server in search server format. The customer then receivesthe rendered HTML of the search.

Time-based Usefulness

As mentioned elsewhere in this application, every aspect of the systemadapts and evolves over time as new observations are made, new log filesare processed, and new patterns emerge. One aspect of this adaptationinvolves giving precedence to usage patterns that occur more recentlyover those that occur in the past. In the preferred implementation ofthe system, recency bias is accomplished through decaying past usagepatterns based on a time decay function. Activeness vectors, forexample, might decay at a rate of 0.01% per day, such that activity thatoccurred in the past has less of an influence on the activeness vectorthan more recent usage. Similarly, term vectors can be set to decay at aspecified rate, such that term to asset associations are biased towardmore recent usage patterns. In this way, the usefulness of assets can becomputed in a time-sensitive manner.

Assets that were useful in the past are not necessarily useful in thepresent. All information stored within the system, including peer scoresand expertise scores, can be set to time decay in a similar fashion.Regarding activeness vectors, assets that are very new or newlyrediscovered may need a boost above and beyond recency bias to enabletheir discovery by the population prior to strong usage patterns havinghad an opportunity to emerge. Thus, very new assets, defined as thoseassets whose very recent activity makes up a large proportion of theirtotal activity over all time, may be given an additional newness bias.It is also possible for an administrator to assign a newness biasexplicitly to certain assets or a collection of assets. This newnessbias makes very new assets appear more active than they are in realityfor a short period of time. It is also possible to identify periodicusage of assets and give activeness biases to assets as they reemerge atspecific times of year, for example.

Usage-based Evaluation of Terms and Phrases

This aspect of the invention relates to relationships amongst terms andamongst terms and phrases that the system infers based on captured usagedata. First, a term affinity (similarity) matrix can be constructed thatrelates terms and phrases to one another. Terms and phrases with highaffinities for one another are considered to be aspects of a singletopic and may even be synonyms for one another. The term affinity matrixcan be constructed based on the frequency of co-occurrence of terms inusers' queries or used links, or by the frequency of co-occurrence ofterms in assets' term vectors, for example. This matrix, in combinationwith linguistic characteristics of the terms and phrases themselves, canbe used to identify synonyms, acronyms, and atomic phrasesautomatically. Atomic phrases are ordered sets of two or more wordswhose frequent occurrence together indicate that they should beconsidered a single multi-word phrase rather than multiple independentwords. The term affinity matrix in combination with navigational usagepatterns and assets' term vectors can even be used to detect terms andphrases that are sub-topics of other terms and phrases. Because all suchidentified relationships between terms/phrases and automatic detectionof synonyms, acronyms, and atomic phrases are based on usage by acommunity of users, identified relationships are inherently tailored toa specific community.

Target Applications for the Invention

In addition to the foregoing discussion, the invention is also useful inmarketing lead generation for business Web sites, sales and channelpartner extranets, customer support sites, vertical healthcareapplications such as physician portals and patient research sites;vertical government applications such as citizen portals; and financialservices and insurance vertical applications such as agent and advisorportals.

Although the invention is described herein with reference to thepreferred embodiment, one skilled in the art will readily appreciatethat other applications may be substituted for those set forth hereinwithout departing from the spirit and scope of the present invention.Accordingly, the invention should only be limited by the Claims includedbelow.

1. A computer implemented method for automatically determining any ofimportance of an on-line asset, topic of said asset, a user's currentcontext/topic of interest, and relationships among assets and queries,without asking members of an on-line community directly comprising aprocessor executing the steps of: observing usage patterns of saidon-line community; employing automatic techniques to extract patternsfrom said usage; learning affinities between and among users, assets,and topics/terms; identifying usefulness of an online asset by observinguser implicit behaviors in connection with said usage patterns of saidonline asset and by extracting behavioral patterns from saidobservations; refining said identified online asset usefulness bycontext, wherein the context of each online asset is automaticallydetected based on observed terms/topics from individual and group userbehaviors when said online asset is determined to be useful based uponsaid individual and group user behaviors; and, given a user query, anyof: determining a user's current context/topic based on the user's queryand refining said identified user context based on previous searchesperformed and assets used during the user's navigation; comparing theidentified context of a current user with the identified context of allassets within the system and assigning to each asset a score based onits similarity to the current user's context; and using said similarityscore to present a user with recommended assets; said observed usagepatterns comprising user online search, navigation, and interactionbehavior, said behavior including any of searches performed and positionin user trail; assets viewed and position in user trail; dwell, range,scrolling, think time, and mouse movement on an asset; anchors and linesused in asset text; virtual bookmarks and virtual printing; and explicitdownloading, emailing, printing, saving, and removing.
 2. The method ofclaim 1, further comprising the step of: using said observed usagepatterns for any of disambiguating search Terms, suggesting new terms,and automatically inserting search terms into user queries to expand asearch based on said terms co-occurrence or lack thereof in identifiedasset context/topics.
 3. The method of claim 1, further comprising thestep of: representing asset topics using a vector-space model and UF-IDF(usage frequency, inverse document frequency) weighting.
 4. The methodof claim 1, further comprising the step of: determining co-occurrence oftopics in assets by representing said terms using a documentvector-space model and UF-IDF (usage frequency, inverse documentfrequency) weighting.
 5. The method of claim 1, further comprising thestep of: classifying usage of a given asset by a given user assuccessful or unsuccessful use based on a combination of factorscomprising any of time spent on an asset, explicit user actions taken onsaid asset, said actions comprising any of print, save, email, andbookmark, mouse and keyboard behavior, length and distribution of stablepauses between activity, navigation to and from said asset, and browserwindow state, said state comprising any of foreground, background,minimized, maximized, and closed.
 6. The method of claim 1, furthercomprising the step of: providing a significantly greater weighting tothe importance of implicit observations over the weighting provided toexplicit observations.
 7. The method of claim 1, further comprising thesteps of: analyzing all observations; and via said analysis, generatinga set of recommendations comprising distilled experiences from on-linecommunity; wherein said recommendations age over time and are discardedif they have relatively little value; and wherein recommendations whichare most valuable based on repeated usage are stored into a long termmemory.
 8. The method of claim 1, further comprising the step of: for agiven user who may be anonymous, said user visiting a particular site,and a given context comprising any of what page the user is on and howthe user got there, providing recommendations to said user that allowsaid user to navigate said site more efficiently.
 9. The method of claim1, further comprising the steps of: generating a set a recommendationsthat may be applied to a search; and for a given user who may beanonymous, and a given search query, using said recommendations torefine and augment a resulting search.
 10. The method of claim 9,further comprising the step of: driving said recommendations not just byindividual uses, but by the use of said on-line community, leveragingthe wisdom of crowds and community emergent behavior.
 11. The method ofclaim 1, further comprising the step of: identifying said on-linecommunity as comprising any of one or more peer groups and/or expertgroups based on an information context; wherein said one or more peergroups and/or export groups are nested and defined by different levelsof contexts.
 12. An apparatus for automatically determining any ofimportance of an on-line asset, topic of said asset, a user's currentcontext/topic of interests, and relationships among assets and queries,without asking members of an on-line community directly, comprising:means for observing usage patterns of said on-line community; aprocessor for employing automatic techniques to extract patterns fromsaid usage; said processor learning affinities between and among users,assets, and topics/terms; identifying usefulness of an online asset byobserving user implicit behaviors in connection with said usage patternsof said online asset and by extracting behavioral patterns from saidobservations; refining said identified online asset usefulness bycontext, wherein the context of each online asset is automaticallydetected based on observed terms/topics from individual and group userbehaviors when said online asset is determined to be useful based uponsaid individual and group user behaviors; and, given a user query, anyof: determining a user's current context/topic based on the user's queryand refining said identified user context based on previous searchesperformed and assets used during the user's navigation; comparing theidentified context of a current user with the identified context of allassets within the system and assigning to each asset a score based onits similarity to the current user's context; and using said similarityscore to present a user with recommended assets; said observed usagepatterns comprising user online search, navigation, and interactionbehavior, said behavior including any of searches performed and positionin user trail; assets viewed and position in user trail; dwell, range,scrolling, think time, and mouse movement on an asset; anchors and linesused in asset text; virtual bookmarks and virtual printing; and explicitdownloading, emailing, printing, saving, and removing.
 13. The apparatusof claim 12, further comprising: means for using said observed usagepatterns for any of disambiguating search terms, suggesting new terms,and automatically inserting search terms into user queries to expand asearch, based on said terms co-occurrence or lack thereof in identifiedasset context/topics.
 14. The apparatus of claim 12, further comprising:means for representing asset topics using a vector-space model andUF-IDF (usage frequency, inverse document frequency) weighting.
 15. Theapparatus of claim 12, further comprising: means for determiningco-occurrence of topics in assets by representing said terms using adocument vector-space model and UF-IDF (usage frequency, inversedocument frequency) weighting.
 16. The apparatus of claim 12, furthercomprising: means for classifying usage of a given asset by a given useras successful or unsuccessful use based on a combination of factorscomprising any of time spent on an asset, explicit user actions taken onsaid asset, said actions comprising any of print, save, email, andbookmark, mouse and keyboard behavior, length and distribution of stablepauses between activity, navigation to and from said asset, and browserwindow state, said state comprising any of foreground, background,minimized, maximized, and closed.
 17. The apparatus of claim 12, furthercomprising: means for providing a significantly greater weighting to theimportance of implicit observations over the weighting provided toexplicit observations.
 18. The apparatus of claim 12, furthercomprising: means for analyzing all observations; and via said analysis,means for generating a set of recommendations comprising distilledexperiences from said on-line community; wherein said recommendationsage over time and are discarded if they have relatively little value;and wherein recommendations which are most valuable based on repeatedusage are stored into a long term memory.
 19. The apparatus of claim 12,further comprising: for a given user who may be anonymous, said uservisiting a particular site, and a given context comprising any of whatpage the user is on and how the user got there, means for providingrecommendations to said user that allow said user to navigate said sitemore efficiently.
 20. The apparatus of claim 12, further comprising:means for generating a set a recommendations that may be applied to asearch; and for a given user who may be anonymous, and a given searchquery, means for using said recommendations to refine and augment aresulting search.
 21. The apparatus of claim 20 further comprising:means for driving said recommendations not just by individual uses, butby the use of said on-line community, leveraging the wisdom of crowdsand community emergent behavior.
 22. The apparatus of claim 12, furthercomprising: means for identifying said on-line community as comprisingany of one or more peer groups and/or expert groups based on aninformation context; wherein said one or more peer groups, and/or expertgroups are nested and defined by different levels of contexts.